real time contextual collective anomaly detection over...
TRANSCRIPT
Real time contextual collective anomaly detection over
multiple data streams
Yexi Jiang, Chunqiu Zeng
School of Computing and
Information Sciences
Florida International University
Miami, FL, USA
{yjian004,
czeng001}@cs.fiu.edu
Jian Xu
School of Computer Science
Technology and Engineering
Nanjing University of Science
and Technology
Nanjing, China
Tao Li
School of Computing and
Information Sciences
Florida International University
Miami, FL, USA
ABSTRACTAnomaly detection has always been a critical and challeng-ing problem in many application areas such as industry,healthcare, environment and finance. This problem becomesmore di�cult in the Big Data era as the data scale increasesdramatically and the type of anomalies gets more compli-cated. In time sensitive applications like real time monitor-ing, data are often fed in streams and anomalies are requiredto be identified online across multiple streams with a shorttime delay. The new data characteristics and analysis re-quirements make existing solutions no longer suitable.
In this paper, we propose a framework to discover a newtype of anomaly called contextual collective anomaly over acollection of data streams in real time. A primary advan-tage of this solution is that it can be seamlessly integratedwith real time monitoring systems to timely and accuratelyidentify the anomalies. Also, the proposed framework is de-signed in a way with a low computational intensity, and isable to handle large scale data streams. To demonstrate thee↵ectiveness and e�ciency of our framework, we empiricallyvalidate it on two real world applications.
1. INTRODUCTIONAnomaly detection is one of the most important tasks in
data-intensive applications such as the healthcare monitor-ing [22], stock analysis [26], disaster management [32], sys-tem anomaly detection [24], and manufacture RFID man-agement [4], etc. In the Big Data era, the aforementionedapplications often require real-time processing. However, ex-isting data processing infrastructures are designed based oninherent non-stream programing paradigm such as MapRe-duce [11], Bulk Synchronous Parallel (BSP) [30], and theirvariations. To reduce the processing delay, these applica-tions have gradually migrated to stream processing engines [27,10, 1, 9]. As the infrastructures have been changed, anoma-lies in these applications are required to be identified online
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from [email protected].
ODD’14, August 24 - 27 2014, New York, NY, USA.
Copyright 2014 ACM 978-1-4503-2998-9/14/08 ...$15.00
http://dx.doi.org/10.1145/2656269.2656271.
across multiple data streams. The new data characteristicsand analysis requirements make existing anomaly detectionsolutions no longer suitable.
1.1 A Motivating Example
Example 1. Figure 1 illustrates the scenario of monitor-ing a 6-node computer cluster, where the x-axis denotes thetime and the y-axis denotes the CPU utilization. The clusterhas been monitored during time [0, t6]. At time t2, a com-puting task has been submitted to the cluster and the clusterfinishes this task at time t4. As shown, two nodes (markedin dashed line) behave di↵erently from the majority duringsome specific time periods. Node 1� has a high CPU utiliza-tion during [t1, t2] and a low CPU utilization during [t3, t4]while node 2� has a medium CPU utilization all the time.These two nodes with their associated abnormal periods areregarded as anomalies. Besides these two obvious anomalies,there are a slight delay on node 3� due to the network delayand a transient fluctuation on node 4� due to some randomfactors. However, they are normal phenomena in distributedsystems and are not regarded as anomalies.
Figure 1: CPU utilization of a computing cluster
Figure 2: Identified anomalies in Example 1 (Thebox lists the IDs of abnormal streams during speci-fied time period)
A quick solution for stream based anomaly detection is toleverage the techniques of complex event processing (CEP) [23,2] by expressing the anomalies detection rules with corre-sponding continuous query statements. This rule-based de-tection method can be applied to the scenarios where theanomaly can be clearly defined. Besides using CEP, sev-eral stream based anomaly detection algorithms have alsobeen proposed. They either focus on identifying contextualanomaly over a collection of stable streams [7] or collectiveanomaly from one stream [3, 25]. These existing methodsare useful in many applications but they still cannot iden-tify certain types of anomalies. A simple example of suchscenario is illustrated in Example 1.
Figure 2 plots the ground truth as well as all the anomaliesidentified by existing methods including the CEP query withthree di↵erent rules (Rule-CQ1, 2, and 3), the collectivebased anomaly detection [6], and contextual based anomalydetection [8].
To detect the anomalies via CEP query, the idea is to cap-ture the events when the CPU utilizations of nodes are toohigh or too low. An example query following the syntax of[2] can be written as follows:
PATTERN SEQ(Observation o[])WHERE avg(o[].cpu) oper threshold(AND|OR avg(o[].cpu) oper threshold)⇤
WITHIN {length of sliding window}
where the selection condition in WHERE clause is the conjunc-tion of one or more boolean expressions, oper is one of {>,<, <>, ==}, and threshold can be replaced by any validexpression. However, CEP queries are unable to correctlyidentify the anomalies in Figure 1 no matter how the se-lection conditions are specified. For instance, setting thecondition as avg(o[].cpu) > {threshold} would miss theanomalies during [t3, t4] (Rule-CQ1); setting the conditionas avg(o[].cpu) < {threshold} would miss the anomaliesduring [t1, t2] (Rule-CQ2); and combining the two above ex-pressions with OR still does not work (Rule-CQ3). Besidesdeciding the selection condition, how to rule out the situa-tions of slight delays and transient fluctuations, and how toset the length of the sliding windows are all di�cult prob-lems when writing the continuous queries. The main reasonis that the continuous query statement is not suitable tocapture the contextual information where the “normal” be-haviors are also dynamic (the utilizations of normal nodesalso change over time in Figure 1).
Compared with CEP based methods, contextual anomalydetection methods (such as [14, 17]) achieve a better ac-curacy as they utilize the contextual information of all thestreams. However, one limitation of contextual based meth-ods is that they do not leverage the temporal informationof streams and are not suitable for anomaly detection indynamic environments. Therefore, these methods wouldwrongly identify the slightly delayed and fluctuated nodesas anomalies.
For the given example, collective anomaly detection meth-ods do not work well neither. This is because these methodswould identify the anomaly of each stream based on its nor-mal behaviors. Once the current behavior of a stream isdi↵erent from its normal behaviors (identified based on his-torical data), it is considered as abnormal. In the example,when the cluster works on the task during [t3, t4], all the
working nodes would be identified as abnormal due to thesudden burst.
1.2 ContributionsIn this paper, we propose an e�cient solution to identify
this special type of anomaly in Example 1, named contextualcollective anomaly. Contextual collective anomalies bear thecharacteristics of both contextual anomalies and collectiveanomalies. This type of anomaly is common in many ap-plications such as system monitoring, environmental mon-itoring, and healthcare monitoring, where data come fromdistributed but homogeneous data sources. We will formallydefine this type of anomaly in Section 2.
Besides proposing an algorithm to discover the contextualcollective anomalies over a collection of data streams, we alsoconsider the scale-out ability of our solution and develop adistributed streaming processing framework for contextualcollective anomaly detection. More concretely, our contri-butions can be described as follows:
• We provide the definition of contextual collective anomalyand propose an incremental algorithm to discover thecontextual collective anomalies in real time. The pro-posed algorithm combines the contextual as well asthe historical information to e↵ectively identify theanomalies.
• We propose a flexible three-stage framework to dis-cover such anomalies from multiple data streams. Thisframework is designed to be distributed and can beused to handle large scale data by scaling out the com-puting resources. Moreover, each component in theframework is pluggable and can be replaced if a bettersolution is proposed in the future.
• We empirically demonstrate the e↵ectiveness and e�-ciency of our solution through the real world scenarioexperiments.
The rest of the paper is organized as follows. Section 2gives a definition of contextual collective anomaly and thenpresents the problem statement. Section 3 provides an overviewof our proposed anomaly detection framework. We intro-duce the three-stage anomaly detection algorithm in detailin Section 4. Section 5 presents the result of experimentalevaluation. The related works are discussed in Section 6.Finally, we conclude in Section 7.
2. PROBLEM STATEMENTIn this section, we first give the notations and definitions
that are relevant to the anomaly detection problem. Then,we formally define the problem based on the given notationsand definitions.
Definition 1. Data Stream. A data stream Si
is anordered infinite sequence of data instances {s
i1, si2, si3, ...}.Each data instance s
it
is the observation of data stream Si
at timestamp t.
The data instances sit
in Si
can have any number of di-mensions, depending on the concrete applications. For theremaining of this paper, the terms “data instance” and “ob-servation” would be used interchangeably. To make the no-tation uncluttered, we use s
i
in places where the absence oftimestamp does not cause the ambiguity.
Definition 2. Stream Collection. A stream collec-tion S = {S1, S2, ..., Sn
} is a collection of data streams. Thenumber of streams |S| = n.
The input of our anomaly detection framework is a streamcollection. For instance, in example 1, the input streamcollection is S = { 1�, 2�, 3�, 4�, 5�, 6�}
Definition 3. Snapshot. A snapshot is a set of key-value pairs S(t) = {S
i
: sit
|Si
2 S}, denoting the set ofthe observations {s1t, s2t, · · · , s|S|t} of the data streams instream collection S at time t.
A snapshot captures the configuration of each data streamin the stream collection for a certain moment. Taking Fig-ure 1 for example, the snapshot at time t5 is S(t5) = { 1� :0%, 2� : 50%, 3� : 0%, 4� : 20%, 5� : 0%, 6� : 0%}. Forsimplicity, we use S(i) to denote the ith dimension of theobservations in a certain snapshot. Note that all the obser-vations in a snapshot have the same dimension.
Definition 4. Contextual Collective Anomaly. Acontextual collective stream anomaly is denoted as a tuple< S
i
, [tb
, te
], N >, where Si
denote a data stream from thecollection of data streams S, [t
b
, te
] is the associated timeperiod when S
i
is observed to constantly deviate from themajority streams in S, and N indicates the severity of theanomaly.
In Example 1, 3 contextual collective anomalies can befound in total. During time period [t1, t2], node 1� be-haves constantly di↵erent from the other nodes, so thereis an anomaly < 1�, [t1, t2], N1>. The other two contextualcollective anomalies, < 1�, [t3, t4], N2> and < 2�, [0, t6], N3>,can also be found with the same reason.In Definition 4, the severity of deviation is measured in
a given metric space M with a distance function f : sit
⇥sku
! R. For simplicity, we use Euclidean distance as anexample throughout this paper.Problem Definition. The anomaly detection problem in
our paper can be described below: Given a stream collectionS = {S1, S2, ..., Sm
}, identify the source of the contextualcollective anomalies S
i
, the associated time period [ts
, te
],as well as a quantification about the confidence of the detec-tion p. Moreover, the detection has to be conducted on datastreams that look-back is not allowed and the anomalies areable to be identified in real time.
3. FRAMEWORK OVERVIEWIn this section, we briefly describe how the aforemen-
tioned problem is addressed and then introduce the proposeddistributed real time anomaly detection framework from ahigh level perspective.As previously mentioned, this paper focuses on discover-
ing contextual collective anomalies over a collection of datastreams obtained from a homogeneous distributed environ-ment. An example of the homogeneous distributed envi-ronment is the system with load balance, which is widelyused at the backend by the popular web sites like Google,Facebook, and Amazon, etc.It is known that, in such a kind of environment, the com-
ponents should behave similar to each other. Therefore, the
snapshots (the current observations) of these streams shouldbe close to each other at any time. Naturally, we need toidentify the anomalies by investigating both contextual in-formation (the information of the current snapshot) and col-lective information (the historical information).
Figure 3: The snapshotat a certain timestamp
Figure 4: Random par-tition of data instance
The anomaly detection is conducted in three stages: thedispatching stage, the scoring stage, and the alert stage. Thefunctionality of the three stages are briefly described as fol-lows:
• Dispatching stage: This stage uses dispatchers to re-ceive the observations from external data sources andthen shu✏e the observations to di↵erent downstreamprocessing components.
• Scoring stage: This stage quantifies the candidate anoma-lies using snapshot scorer and then stream scorer.
The snapshot scorer leverages contextual informationto quantify the confidence of anomaly for each datainstance at a given snapshot. Taking Figure 3 for ex-ample, it shows the data distribution by taking thesnapshot of the 2-dimensional data instances of 500streams at timestamp t. As shown, most of the datainstances are close to each other and located in a densearea. These data instances are not likely to be identi-fied as anomalies as their instance anomaly scores aresmall. On the contrary, a small portion of the data in-stances (those points that are far away from the denseregion) have larger instance anomaly scores and aremore likely to be abnormal.
A data instance with a high anomaly score does notindisputably indicate its corresponding stream to bea real anomaly. This is because the transient fluctu-ation and phase shift are common in real world dis-tributed environment. To mitigate such e↵ects, thestream scorer is designed to handle the problem. Inparticular, the stream scorer combines the informationobtained from the instance scorer and the historicalinformation of each stream to quantify the anomalyconfidence of each stream.
• Alert stage: The alert stage contains the alter trigger.The alert triggers leverage the unsupervised learningmethods to identify and report the outliers.
The advantage of our framework is reflected by the ease ofintegration, the flexibility, and the algorithm independence.
Firstly, any external data sources can be easily fed to theframework for anomaly detection. Moreover, the compo-nents in every stage can be scaled-out to increase the pro-cessing capability if necessary. The number of componentsin each stage can be easily customized according to the datascale of concrete applications. Furthermore, the algorithmsin each stage can be replaced and upgraded with better alter-natives and the replacement would not interfere with otherstages.
4. METHODOLOGY
4.1 Data Receiving and DispatchingThe dispatching stage is an auxiliary stage in our frame-
work. When the data scale (i.e., the number of streams)is too large for a single computing component to process,the dispatcher would shu✏e the received observations todownstream computing components in the scoring stage (asshown in Figure 4). By leveraging random shu✏ing algo-rithm like Fisher-Yates shu✏e [12], dispatching can be con-ducted in constant time per observation. After dispatching,each downstream component would conduct scoring inde-pendently on a sampled stream observations with identicaldistribution.
Ideally, the observations coming from homogeneous datasources have very similar measurable value (e.g. workload ofeach server in a load balanced system), but random factorscan easily cause the variations of the actual observations.Therefore in fact, an observation si 2 Rd is viewed as theideal case value sideal with additive Gaussian noise so thatsi = sideal + ✏, ✏ ⇠ N (µ,⌃). For those data sources inabnormal conditions, their observations are generated ac-cording to a di↵erent but unknown distributed. It is notdi�cult to know that, given enough observations, the meanand covariance can be easily estimated locally via maxi-
mum likelihood, i.e. µ̂ML
=P
i
sin
and ccov(S(x), S(y))ML
=1
n�1
Pn
i=1(si(x)� S̄(x))(si
(y)� S̄(y)), where S̄(x) and S̄(y)respectively denote the sample mean of dimension x and y.
4.2 Snapshot Anomaly QuantificationQuantifying the anomaly in a snapshot is the first task
in the scoring stage and we leverage snapshot scorer in thisstep. This score measures the amount of deviation of thespecified observation s
it
to the center of all the observationsin a snapshot S(t) at timestamp t.
To quantify the seriousness of snapshot anomaly, we pro-pose a simple yet e�cient method. The basic idea is that theanomaly score of an observation is quantified as the amountof uncertainty it brings to the snapshot S(t). As the observa-tions in a snapshot follows the normal distribution, it is suit-able to use the increase of entropy to measure the anomalyof an observation. To quantify the anomaly score, two typesof variance are needed: the variance and the leave-one-outvariance, where the leave-one-out variance is the varianceof the distribution when one specific data instance is notcounted.
A naive algorithm to quantify the anomaly scores requiresquadratic time (O(dn+ dn2)). By reusing the intermediateresults, we propose an improved algorithm with time com-plexity linear to the number of streams. The pseudo code ofthe proposed algorithm is shown in Algorithm 1. As illus-trated, matrix M is used to store the distances between each
Algorithm 1 Snapshot Anomaly Quantification
1. INPUT: Snapshot S(t) = (s1, · · · , sn), where si
2 Rd.2. OUTPUT: Snapshot anomaly scores N 2 Rn.3. Create a d⇥ n matrix M = (s1, · · · , sn)4. Conduct 0-1 normalization on rows of M .5. x 0d and N = 0d
6. m (E(S(1)), · · · ,E(S(d))T7. M (s1t �m, s2t �m, · · · , s
dt
�m)8. for j 0 to d do9. x
j
= ||jth column of M ||2210. end for11. for all s
i
2 S(t) do12. Calculate N
i
according to Equation (1).13. end for14. Conduct 0-1 normalization on N .15. return N
dimension of the observations to the corresponding mean.Making use of M , the leave-one-out variance can be quickly
calculated as �2ik
=n�
2k
�M
2i,k
n�1 , where �2k
denotes the variance
of dimension k and �2ik
denotes the leave-one-out varianceof dimension k by excluding s
i
. As the entropy of normaldistribution is H = 1
2 ln(2⇡e�2), the increase of entropy forobservation s
i
at dimension k can be calculated as
dk
= H 0k
�Hk
= ln�2ik
�2k
= ln(x
k
�M2ik
)/(n� 1)x
j
/n, (1)
where H 0k
and Hk
respectively denote the entropy of thesnapshot distribution if s
i
is not counted or counted.Summing up all dimensions, the snapshot anomaly score
of sit
is Nit
=P
k
dk
. Note that the computation implicitlyignores the correlation between dimensions. The reason isthat if an observation is an outlier, the correlation e↵ectwould only deviate it further from other observations.
4.3 Stream Anomaly Quantification
As a stream is continuously evolving and its observationsonly reflect the transient behavior, snapshot anomaly scorealone would result in a lot of false-positives due to the tran-sient fluctuation and slight phase shift. To mitigate suchsituations, it is critical to quantify the stream anomaly byincorporating the historical information of the stream.
An intuitive way to solve this problem is to calculate thestream anomaly score from the recent historical instancesstored in a sliding window. However, this solution has twoobvious limitations: (1) It is hard to decide the windowlength. A long sliding window would miss the real anomalywhile a short sliding window cannot rule out the false-positives.(2) It ignores the impact of observations that are not in thesliding window. The observation that is just popped outfrom the sliding window would immediately and totally loseits impact to the stream.
To well balance the history and the current observation,we use stream anomaly score N
i
to quantify how significanta stream S
i
behaves di↵erently from the majority of thestreams. To quantify N
i
, we exploit the exponential decayfunction to control the influence depreciation. Supposing�t is the time gap between two adjacent observations, theinfluence of an observation s
it
at timestamp tx+k
= tx
+k�t can be expressed as N
it
x
(tx+k
) = Nit
x
(tx
+ k�t) =
Nit
x
e��kt, (� > 0), where � is a parameter to control thedecay speed. In the experiment evaluation, we will discusshow this parameter a↵ects the anomaly detection results.
To make the notation uncluttered, we use t�i
to denotethe timestamp that is i�t ahead of current timestamp t, i.e.t�i
= t�i�t. Summing up the influences of all the historicalobservations, the overall historical influence I
it
for currenttimestamp t can be expressed as Equation (2).
Iit
= Nit�1(t) +N
it�2(t) +Nit�3(t) + ...
= Nit�1e
�� +Nit�2e
�2� +Nit�3e
�3� + ...
= e��(Nit�1 + e��(N
it�2 + e��(Nit�3 + ...
= e��(Nit�1 + I
it�1).
(2)
The stream anomaly score of stream Si
is the summation ofthe data instance anomaly score of current observation N
it
and the overall historical influence, i.e.,
Ni
= Nit
+ Iit
. (3)
As shown in Equation (2), the overall historical influencecan be incrementally updated with cost O(1) for both timeand space complexity. Therefore, stream anomaly scorer canbe e�ciently computed.
4.3.1 Properties of Stream Anomaly ScoreThe properties of stream anomaly score make our frame-
work insensitive to the transient fluctuation and e↵ective tocapture the real anomaly.
Comparing to the transient fluctuation, the real anomalyis more durable. Figure 5 shows the situations of a transientfluctuation (in the left subfigure) and a real anomaly(in theright subfigure). In both situations, the stream behaves nor-mally before timestamp t
x
. For the left situation, a transientfluctuation occurs at timestamp t
x+1, and then the streamreturns to normal at timestamp t
x+2. For the right situa-tion, an anomaly begins at timestamp t
x+1, lasts for a whiletill timestamp t
x+k
, and then the stream returns to nor-mal afterwards. Based on Figure 5, we show two propertiesabout the stream anomaly score.
Figure 5: Transient Fluctuation and Anomaly
Property 1. The increase of stream anomaly score causedby transient disturbance would decrease over time.
Property 2. The increase of stream anomaly score causedby anomaly would be accumulated over time.
Similar properties can also be shown for the situation ofslight shifts. A slight shift can be treated as two transientfluctuations occur at the beginning and the end of the shift.In the next section, we will leverage these two properties toe↵ectively identify the anomalies in the Alert Stage.
4.4 Alert TriggeringMost of the stream anomaly detection solutions [13] iden-
tify the anomalies by picking the streams with top-k anomalyscores or the ones whose scores exceed a predefined thresh-old. However, these two approaches are not practical in real
world applications for the following reasons: (1) Thresholdis hard to set. It requires the users to understand the un-derlying mechanism of the application to correctly set theparameter. (2) The number of anomalies are changing allthe time. It is possible that more than k anomaly streamsexist at one time, then the top-k approach would miss thesereal anomalies.
Figure 6: Abnormal Streams Identification
To eliminate the parameters, we propose an unsupervisedmethod to identify and quantify the anomalies by leverag-ing the distribution of the anomaly scores. The first step isto find the median of the stream anomaly scores (Nmedian).If the distance between a stream anomaly score and themedian score is larger than the distance between the me-dian score and the minimal score (Nmin), the correspond-ing stream is regarded as abnormal. As shown in Figure 6,this method implicitly defines a dynamic threshold (shownas the dashed line) based on the hypothesis that there isno anomaly. If there is no anomaly, the skewness of theanomaly score distribution should be small and the medianscore should be close to the mean score. If the hypothesis istrue, Nmedian �Nmin should be close to half of the distancebetween the minimum score and the maximum score. On thecontrary, if a score N
i
is larger than 2 ⇥ (Nmedian � Nmin),the hypothesis is violated and all the streams with scores atleast N
i
are abnormal.Besides the general case, we also need to handle one spe-
cial case: a transient fluctuation occurs at the current times-tamp. According to Property (1) in Section 4.3.1, the e↵ectof transient fluctuation is at most d
upper
= Nit
x+1 �Njt
x+1
and it will monotonically decrease. Therefore, even a streamwhose anomaly score is larger than 2⇥ (Nmedian �Nmin), itcan still be a normal stream if the di↵erence between itsanomaly score and Nmin is smaller than d
upper
. To prunethe false-positive situations caused by transient fluctuation,the stream is instead identified as abnormal if
Ni
> max(2(Nmedian �Nmin), Nmin + dupper
). (4)
Another thing needs to be noted is that the stream anomalyscores have a upper bound
d
upper
1�e
��
. According to the prop-erty of convergent sequence, the stream anomaly scores of allstreams would converge to this upper bound. When the val-ues of stream anomaly scores are close to the upper bound,they tend to be close to each other and hard to be distin-guished. To handle this problem, we reset all the streamanomaly scores to 0 whenever one of them close to the up-per bound.
In terms of the time complexity, the abnormal streams canbe found in O(n) time. Algorithm 2 illustrates the algorithmof stream anomalies identification. The median of the scorescan be found in O(n) in the worst case using the BFPRTalgorithm [5]. Besides finding the median, this algorithmalso partially sorts the list by moving smaller scores beforethe median and larger scores after the median, making ittrivial to identify the abnormal streams by only checkingthe streams appearing after the median.
5. EXPERIMENTAL EVALUATION
Algorithm 2 Stream Anomaly Identification
1. INPUT: �, and unordered stream profile list S ={S1, ..., Sn
}.2. mIdx d |S|
2 e3. Nmedian BFPRT(S, mIdx )4. Nmin min(S
i
.score|0 i mIdx)5. N
max
Nmedian
6. for i mIdx to |S| do7. if Condition (4) is satisfied then8. Trigger alert for S
i
with score Ni
at current time.9. if N
i
> Nmax
then10. N
max
Ni
11. end if12. end if13. end for14. if N
max
is close to the upper bound then15. Reset all stream anomaly scores.16. end if
To investigate the e↵ectiveness and e�ciency of our frame-work, we design several sets of experiments with one realworld data applications: anomaly detection over a comput-ing cluster. Taking these two applications as case studies, weshow that our proposed framework can e↵ectively identifythe abnormal behavior of streams. It should be pointed outthat our proposed framework can also be applied to manyother application areas such as PM2.5 environment monitor-ing, healthcare monitoring, and stock market monitoring.
System anomaly detection is one of the critical tasks insystem management. In this set of experiments, we showthat our proposed framework can e↵ectively and e�cientlydiscover the abnormal behaviors of the computer nodes withhigh precision and low latency.
5.1 Experiment SettingsFor the experiments, we leverage a distributed system
monitoring tool [31] into a 16-node computing cluster. Thenwe deploy the proposed anomaly detection program on anexternal computer to analyze the collected trace data in realtime. To well evaluate our proposed framework, we termi-nate all the irrelevant processes running on these nodes. Onthis node, we intentionally inject various types of anoma-lies and monitor their running status for 1000 seconds. Thesource code of injection program is available at https://
github.com/yxjiang/system-noiser. The details of the in-jections are listed in Table 1.
Table 1: List of InjectionsNo. Time Period Node Description
1 [100, 150] 2 Keep CPU utilization above 95.2 [300, 400] 3 Keep memory usage at 70%.3 [350, 400] 3 Keep CPU utilization above 95%.4 [600, 650] 4 Keep memory usage at 70%.5 [900, 950] 2,5 Keep CPU utilization above 95%.6 [800, 850] 1-5,7-16 Keep CPU utilization above 95%.
Through these injections, we can answer the followingquestions about our framework: (1) Whether our frame-work can identify the anomalies with di↵erent types of rootcauses. (2) Whether our framework can identify multipleanomalies occurring simultaneously.
5.2 Results Analysis
Figure 7: Injections and the captured alerts
Figure 7 illustrates the results of this experiment by plot-ting the actual injections (top 6 sub-figures) as well as thecaptured alerts (the bottom subplots), where the x-axis rep-resents the time and y-axis represents the idled CPU uti-lization, idle memory usage or the number of anomalies ineach timestamp. We evaluate the framework from 5 aspectsthrough carefully-designed injections.
1. Single dimension (e.g. idle CPU utilization or idlememory usage) of a single stream behaves abnormally.This is the simplest type of anomalies. It is generatedby injections No.1 and No.4 in Table 1. As shownin Figure 7, our framework e↵ectively identifies theseanomalies with the correct time periods.
2. Multiple dimensions (e.g. CPU utilization and mem-ory usage) of a single stream behaves abnormally at thesame time. This type of anomalies is generated by in-jections No.2 and No.3 in Table 1, and our frameworkcorrectly captures such anomalies during the time pe-riod [300, 400]. One thing should be noted is that thestream anomaly score of node 3 increases faster duringthe time period [350, 400] than the time period [300,350]. This is because two types of anomalies (CPUutilization and memory usage) appear simultaneouslyduring the time period [350, 400].
3. Multiple streams behave abnormally simultaneously. Thistype of anomalies is generated by injection No.5. Dur-ing the injection time period, our framework correctlyidentifies both anomalies (on node 2 and node 5).
4. Stable but abnormal streams. This kind of anomaly isindirectly generated by injection No.6 in Table 1. Thisinjection emulates the scenario that all the nodes butone (i.e., node 6) in a cluster received the commandof executing a task. As is shown, although the CPUutilization of node 6 behaves stable all the time, it isstill considered to be abnormal during the time period[800, 850]. This is because it remains idle when all theother nodes are busy.
5. Transient fluctuation and slight delay would not causefalse-positive. As this experiment is conducted in adistributed environment, delays exist and vary for dif-ferent nodes when executing the injections. Despite
this intervention, our framework still does not reporttransient fluctuations and slight delays as anomalies.
Based on the evaluation results, we find that our solutionis able to correctly identify all the anomalies in all these 5di↵erent cases.
5.3 Results ComparisonTo demonstrate the superiority of our framework, we also
conduct experiments to identify the anomalies with the sameinjection settings using the alternative methods includingcontextual anomaly detection (CAD) and rule-based contin-uous query (Rule-CQ). The contextual anomaly detectionis equivalent to the snapshot scoring in our framework. Forthe rule-based continuous query, we define three rules to cap-ture three types of anomalies, including high CPU utilization(rule 1), low CPU utilization (rule 2), and high memory us-age anomalies (rule 3), respectively. Di↵erent combinationsof the three rules are used in the experiments.
Figure 8: Generated alerts by CAD and Rule-CQ
The generated alerts of these methods are shown in Fig-ure 8, where the x-axis denotes the time and y-axis denotesthe number of anomalies. As illustrated, the contextualanomaly detection method generates a lot of false alerts.This is because this method is sensitive to the transientfluctuation. Once an observation deviates from the othersat a timestamp, an alert would be triggered. For Rule-CQmethod, we experiment all the combinations and report theresults of the two best combinations: C1 (rule 1 or rule 2)and C2 (rule 2 or rule 3). Similarly, the Rule-CQ methodalso generates many false alerts since it is di�cult to userules to cover all the anomaly situations. Table 2 quanti-tatively shows the precision, recall, and F-measure of thethree methods as well as the results of our method. Thelow-precision and high-recall results of CAD and Rule-CQindicate that all these method are too sensitive to fluctua-tions.
Table 2: Measures of di↵erent methods performedon the trace data in distributed environment``````````̀Measure
Methodprecision recall F-measure
CAD 0.4207 1.0000 0.5922C1: Rule 1||3 0.5381 1.0000 0.6997C2: Rule 2||3 0.0469 1.0000 0.0897
Our method (worst case) 0.9832 0.8400 0.9060
5.4 A real system problem detectedWe have identified a real system problem when deployed
our framework on two computing clusters in our department.In one of the clusters, we continuously receive alerts. Log-ging into the cluster, we find the CPU utilization is high
even no tasks are running. We further identify that the highCPU utilization is caused by several processes named hfsd.We reported the anomaly to IT support sta↵s and they con-firmed that there exist some problems in this cluster. Thehigh CPU utilization is caused by continuous attempts toconnect to a failure node in the network file system. Afterfixing this problem, these out-of-expectation but real alertsdisappear.
6. RELATED WORKSAlthough anomaly detection has been studied for years [15,
8]. To the best of our knowledge, our method is the firstone that focused on mining contextual collective anomaliesamong multiple streams in real time. In this section, webriefly review two closely related areas: mining outliers fromdata streams and mining outliers from trajectories.
With the emerging requirements of mining data streams,several techniques have been proposed to handle the dataincrementally [33, 20, 19, 28, 18]. Pokrajac et al. [25] mod-ified the static Local Outlier Factor (LOF) [6] method asan incremental algorithm, and then applied it to find datainstance anomalies from the data stream. Takeuchi and Ya-manishi [16] trained a probabilistic model with an onlinediscounting learning algorithm, and then use the trainingmodel to identify the data instance anomalies. Angiulli andFassetti [3] proposed a distance-based outlier detection al-gorithm to find the data instance anomalies over the datastream. However, all the aforementioned works focused onthe anomaly detection of a single stream, while our work isdesigned to discover the contextual collective anomalies overmultiple data streams.
A lot of works have been conducted on trajectory outlierdetection. One of the representative work on trajectory out-lier detection is conducted by Lee et al. [21]. They proposeda partition-and-detection framework to identify the anomalysub-trajectory via the distance-based measurement. Lianget al. [29] improved the e�ciency of Lee’s work by only com-puting the distances among the sub-trajectories in the samegrid. As the aforementioned two algorithms require to ac-cess the entire dataset, they cannot be adapted to trajectorystreams. To address the limitation, Bu et al. [7] proposeda novel framework to detect anomalies over continuous tra-jectory streams. They built local clusters for trajectoriesand leveraged e�cient pruning strategies as well as indexingto reduce the computational cost. However, their approachidentified anomalies based on the local-continuity propertyof the trajectory, while our method does not make such anassumption. Our approach is close to the work of Ge etal. [13], where they proposed an incremental approach tomaintain the top-K evolving trajectories for tra�c moni-toring. However, their approach mainly focused on the geo-spatial data instances and ignored the temporal correlations,while our approach explicitly considers the temporal infor-mation of the data instances.
7. CONCLUSIONIn this paper, we propose a real time anomaly detection
framework to identify the contextual collective anomaliesfrom a collection of streams. Our proposed method firstlyquantifies the snapshot level anomaly of each stream basedon the contextual information. Then the contextual infor-mation and the historical information are used in combina-
tion to quantify the anomaly severity of each stream. Basedon the distribution of the stream anomaly scores, an implicitthreshold is dynamically calculated and the alerts are trig-gered accordingly. To demonstrate the usefulness of the pro-posed framework, several sets of experiments are conductedto demonstrate its e↵ectiveness and e�ciency.
8. ACKNOWLEDGEMENTThe work was supported in part by the National Sci-
ence Foundation under grants DBI-0850203, HRD-0833093,CNS-1126619, and IIS-1213026, the U.S. Department of Home-land Security under grant Award Number 2010-ST-06200039,Army Research O�ce under grant number W911NF-10-1-0366 andW911NF-12-1-0431, National Natural Science Foun-dation of China under grant number 61300053, and an FIUDissertation Year Fellowship.
9. REFERENCES[1] A.Arasu, B.Babcock, S.Babu, M.Datar, K.Ito,
I.Nizhizawa, J.Rosenstein, and J.Widom. Stream: Thestanford stream data manager. In SIGMOD, 2003.
[2] J. Agrawal, Y. Diao, D. Gyllstrom, and N. Immerman.E�cient pattern matching over event streams. InSIGMOD, 2008.
[3] F. Anguilli and F. Fassetti. Detecting distance-basedoutliers in streams of data. In CIKM, 2007.
[4] Y. Bai, F. Wang, P. Liu, C. Zaniolo, and S. Liu. Rfiddata processing with a data stream query language. InICDE, 2007.
[5] M. Blum, R. Floyd, V.Pratt, R.Rivest, and R. Tarjan.Time bounds for selection. Journal of ComputerSystem Science, 1973.
[6] M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander.Lof: Identifying density-based local outliers. InSIGMOD, 2000.
[7] Y. Bu, L. Chen, A. W.-C. Fu, and D. Liu. E�cientanomaly monitoring over moving object trajectorystreams. In KDD, 2009.
[8] V. Chandola, A. Banerjee, and V. Kumar. Anomalydetection: A survey. ACM Computing Surveys, 2009.
[9] S. Chandrasekaran, O.Cooper, A.Deshpande,M.J.Franklin, J.M.Hellerstein, W.Hong,S.Krishnamurthy, S.R.Madden, V.Raman, F.Reiss,and M.A.Shah. Telegraphcq: Continous dataflowprocessing for an uncertain world. In CIDR, 2003.
[10] D.Carney, U.Cetintemel, M.Cherniack, C. Convey,S.Lee, G.Seidman, M.Stonebraker, N.Tatbul, andS.Zdonik. Monitoring streams: A new class of datamanagement applications. In VLDB, 2002.
[11] J. Dean and S. Ghemawat. Mapreduce: Simplifieddata processing on large clusters. In OSDI, 2004.
[12] R. A. Fisher, F. Yates, et al. Statistical tables forbiological, agricultural and medical research.Statistical tables for biological, agricultural andmedical research., (Ed. 3.), 1949.
[13] Y. Ge, H. Xiong, Z.-H. Zhou, H. Ozdemir, J. Yu, andK. C. Lee. Top-eye: Top-k evolving trajectory outlierdetection. In CIKM, 2010.
[14] M. Gupta, A. B. Sharma, H. Chen, and G. Jiang.Context-aware time series anomaly detection forcomplex systems. In WORKSHOP NOTES, page 14,2013.
[15] V. Hodge and J. Austin. A survey of outlier detectionmethodologies. Artificial Intelligence Review, 2004.
[16] J. ichi Takeuchi and K. Yamanishi. A unifyingframework for detecting outliers and change pointsfrom time series. IEEE Transactions on Knowledgeand Data Engineering, 2006.
[17] G. Jiang, H. Chen, and K. Yoshihira. Modeling andtracking of transaction flow dynamics for faultdetection in complex systems. Dependable and SecureComputing, IEEE Transactions on, 3(4):312–326,2006.
[18] Y. Jiang, C.-S. Perng, T. Li, and R. Chang. Asap: Aself-adaptive prediction system for instant cloudresource demand provisioning. In ICDM, 2011.
[19] Y. Jiang, C.-S. Perng, T. Li, and R. Chang. Intelligentcloud capacity management. In NOMS, 2012.
[20] Y. Jiang, C.-S. Perng, T. Li, and R. Chang.Self-adaptive cloud capacity planning. In SCC, 2012.
[21] J.-G. Lee, J. Han, and X. Li. Trajectory outlierdetection: A partition-and-detect framework. InICDE, 2008.
[22] B. Lo, S. Thiemjarus, R. king, and G. Yang. Bodysensor network-a wireless sensor platform for pervasivehealthcare monitoring. In PERVASIVE, 2005.
[23] B. Mozafari, K. Zeng, and C. Zaniolo.High-performance complex event processing over xmlstreams. In SIGMOD, 2012.
[24] A. Oliner, A. Ganapathi, and W. Xu. Advances andchallenges in log analysis. Comm. of ACM, 2012.
[25] D. Pokrajac, A. Lazarevic, and L. J. Latecki.Incremental local outlier detection for data streams. InCIDM, 2007.
[26] Y. Song and L. Cao. Graph-based coupled behavioranalysis: a case study on detecing collaborativemanipulations in stock markets. In IEEE worldcongress on computational intelligence, 2012.
[27] storm. http://storm-project.net.[28] L. Tang, C. Tang, L. Duan, Y. Jiang, C. Zeng, and
J. Zhu. Movstream: An e�cient algorithm formonitoring clusters evolving in data streams. In Grc,2008.
[29] L. Tang, C. Tang, Y. Jiang, C. Li, L. Duan, C. Zeng,and K. Xu. Troadgrid: An e�cient trajectory outlierdetection algorithm with grid-based space division. InNDBC, 2008.
[30] L. G. Valiant. A bridging model for parallelcomputation. Communications of the ACM,33(8):103–111, 1990.
[31] C. Zeng, Y. Jiang, L. Zheng, J. Li, L. Li, H. Li,C. Shen, W. Zhou, T. Li, B. Duan, et al. Fiu-miner: afast, integrated, and user-friendly system for datamining in distributed environment. In KDD, pages1506–1509. ACM, 2013.
[32] L. Zheng, C. Shen, L. Tang, T. Li, S. Luis, S.-C. Chen,and V. Hristidis. Using data mining techniques toaddress critical information exchange needs in disastera↵ected public-private networks. In KDD, 2010.
[33] Y. Zhu and D. Shasha. Statstream: Statisticalmonitoring of thousands of data streams in real time.In VLDB, 2002.