discussion for anomaly & prediction engine

Fujitsu Standard Tool

Copyright 2016 FUJITSU LIMITEDDiscussion for Anomaly & Prediction Engine04 Feb. 2016Hisashi Osanai

0

Copyright 2016 FUJITSU LIMITED0

AgendaCopyright 2016 FUJITSU LIMITEDPOC IntroductionPOC DemoSystem ConfigurationParallel distributed processing platformEx. Batch process / Stream processFindings/Problems from POCWhy Im interested in Monasca Current Concerns and Approach1


POC DemoCopyright 2016 FUJITSU LIMITED

2


Copyright 2016 FUJITSU LIMITEDDemo System ConfigurationMaster serverVisualization serverOSElasticSearchApache(httpd)KibanaJDKOScollection/storedefinitionHadoopSparkfluentdRabbitMQParallel distributed processing platformprocess definitionStream processSparkStreaming/SparkSQLDataconverterTarget server#1OSfluentdcollection definitionfluentdcollection/store definitionSlave server#3SparkOSHadoopJDKJDK

Data collection targetSlave server#2SparkOSHadoopJDKSlave server#1SparkOSHadoopJDKBatchprocessTaskcontrollerTarget server#2OSfluentdcollection definitionTarget server#nOSfluentdcollection definition3


Copyright 2016 FUJITSU LIMITEDParallel distributed processing platformApache Spark(Core)SparkSQL(SQL query)SparkStreaming(Event stream processing)

Parallel distributed processing platformJob Definition(XML)RabbitMQ(Messagebroker)Fluentd(Data collector)HDFS(Distributed File System )ElasticSearch(Real time search engine)Kibana(Data visualization)

Stream data receptionData process with SQLCreate time-series dataAnalysis processEx. stream data analysis in the anomaly detection process

Enable to execute Stream process and Batch process Fast-acting data conversion based on XML-based Job Definition4


Copyright 2016 FUJITSU LIMITEDEx. Batch processParallel distributed processing platformJob definition (XML)TASK:1Read master dataSparkBatchApplicationTASK:2Read Web access logWeb access log

Analysis TASK:3Query and Save

Spark Cluster

HDFS

HDFSAnalyze a lot of Web access log on file system5


Copyright 2016 FUJITSU LIMITEDEx. Stream processParallel distributed processing platformJob definition (XML)RabbitMQReceiverRabbitMQTASK:1Process and store the CPU information

HDFSSparkStreamingApplicationTASK:2Process and store the MEM information

Analysis

Target server

Analyze statistics information (CPU/MEM) in real-time6


Copyright 2016 FUJITSU LIMITEDFindings/Problems from POCNeeds manpower for data collection on target servers Have discussions with customers to define collecting data and then configure fluentd agents (Num of POCs is limited)

Difficult to store experiences of IT analytics Data and its format are different each customer so suitable anomaly detection libraries are also different

Difficult to catch up for anomaly detection librariesRapid tech evolution for Machine Learning such as Mllib, TensorFlow, CNTK and so on 7


Copyright 2016 FUJITSU LIMITEDSeems to solve two problems from POCNeeds manpower for data collection on target servers Monasca provides agents for OpenStack env so we just use them.Difficult to store experiences of IT analytics Data come from Monasca agents and the format is stable. So we use the data as stable input and are looking for which libraries are suitable for this env which is monitored by Monasca

Add a catching function to MonascaBoosts Monasca salesA lot of our customers are interested in IT analyticsFujitsu sells Monasca-based product Why Im interested in Monasca8


Copyright 2016 FUJITSU LIMITEDCurrent ConcernsPerformance for real time anomaly detection (Storm vs. ApacheStreaming)Rapid tech evolution for Machine Learning (Needs to have plugin arch for the libraries)

Approach (a base for discussion)How to move Anomaly & Prediction Engine (APE) dev ahead?IdeaFirst Rebase current prototype on Monasca master (If possible, I would like to do this with Rolands help)Then use it to find out problems

Current Concerns & Approach9



discussion for anomaly & prediction engine

Software