nsa for enterprises log analysis use cases
DESCRIPTION
TRANSCRIPT
![Page 1: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/1.jpg)
VP Developer Evangelism Samisa Abeysinghe
Big Brother for Enterprises: Log Analysis Use Cases
19 Feb 2014
![Page 2: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/2.jpg)
2
About the Presenter
๏ Samisa Abeysinghe VP Developer Evangelism [email protected]
๏ Samisa Abeysinghe, Vice President of Developer Evangelism joined the company in September 2005. Prior to the current role, Samisa used to be VP of Engineering and managed the development of WSO2 Carbon based product plaOorm.
![Page 3: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/3.jpg)
3
About WSO2 ๏ Global enterprise, founded in 2005
by acknowledged leaders in XML, web services technologies, standards and open source
๏ Provides only open source plaOorm-‐as-‐a-‐service for private, public and hybrid cloud deployments
๏ All WSO2 products are 100% open source and released under the Apache License Version 2.0.
๏ Is an AcXve Member of OASIS, Cloud Security Alliance, OSGi Alliance, AMQP Working Group, OpenID FoundaXon and W3C.
๏ Driven by InnovaXon
๏ Launched first open source API Management soluXon in 2012
๏ Launched App Factory in 2Q 2013
๏ Launched Enterprise Store and first open source Mobile soluXon in 4Q 2013
![Page 4: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/4.jpg)
4
What WSO2 Deliver
![Page 5: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/5.jpg)
NSA like Monitoring for Your Enterprise ๏ Analyze volumes of data
๏ Address correlation complexities in analytics
๏ Off line vs Real time operations implications
๏ Some operations got to be in real time, else the value is lost
๏ Summarized data over time (and other) dimensions for analytics
A “Big Brother” that keeps an eye on the whole enterprise
5
![Page 6: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/6.jpg)
Why should I bother? ๏ Deal with high volume (terra bites) of information
๏ In order to make decisions
๏ Real time & Offline
๏ Take action
6
![Page 7: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/7.jpg)
WSO2 :ONLY COMPLETE & INTEGRATED Platform ๏ Complete and integrated for
๏ Data capture
๏ Analysis: both real-time and batch
๏ Visualization
๏ Action taking business process execution
Making data driven intelligence for your enterprise easy
7
![Page 8: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/8.jpg)
WSO2 Big Data Analy0cs Pla4orm for Your Enterprise
WSO2 Big Data Analytics Platform for Your Enterprise
8
![Page 9: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/9.jpg)
Key Elements
Data CollecXon
Data Analysis
Data VisualizaXon
Taking AcXon
9
![Page 10: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/10.jpg)
Use Case Scenario 1: Monitor your Java ApplicaXon System Logs
with BAM & CEP
10
![Page 11: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/11.jpg)
11
& WSO2 CEP
![Page 12: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/12.jpg)
Overview of SoluXon o Send Log Events to
o Business Activity Monitor (BAM) & o Complex Event Processor (CEP)
o Real time Log Event Processing o With CEP
o Batch Processing of Log Data o With BAM analytics
o Visualization of Log Data o With Gadgets on Dashboards
12
![Page 13: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/13.jpg)
Log Event Publishing (BAM)
13
![Page 14: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/14.jpg)
Event Streams & AlerXng (CEP)
14
WSO2 CEP
![Page 15: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/15.jpg)
LogEvent Stream q Meta Data
q clientType {String} q Meta Data
q tenantID {String} q ServerName {String} q appName {String} q logTime {Long} q priority {Long} q message {String} q logger {String} q ip {String} q instance {String} q stacktrace {String}
15
![Page 16: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/16.jpg)
CEP Query
from LogEvents [priority == "ERROR"]
select message, stacktrace, serverName insert into ExceptionStream
Email Body Error Occurred in {{serverName}} – {{message}} {{stacktrace}}
16
![Page 17: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/17.jpg)
AnalyXcs & Batch Processing (BAM)
17
![Page 18: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/18.jpg)
Hive Query CREATE EXTERNAL TABLE IF NOT EXISTS LogEventInfo (key STRING, tenantID INT,serverName STRING, appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING) STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ( "cassandra.host" = "localhost", "cassandra.port" = "9160","cassandra.ks.name" = "EVENT_KS", "cassandra.ks.username" = "admin","cassandra.ks.password" = "admin", "cassandra.cf.name" = "log_0_AS_2014_01_23", "cassandra.columns.mapping" = ":key,payload_tenantID,payload_serverName,payload_appName, payload_priority,payload_logTime,payload_logger,payload_message" ); CREATE EXTERNAL TABLE IF NOT EXISTS Logs(tenantID INT,serverName STRING, appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING) STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES ( 'mapred.jdbc.driver.class' = 'com.mysql.jdbc.Driver', 'mapred.jdbc.url' = 'jdbc:mysql://localhost:3306/MYBAMDB', 'mapred.jdbc.username' = 'root','mapred.jdbc.password' = 'root', 'hive.jdbc.update.on.duplicate' = 'true', 'hive.jdbc.table.create.query' = 'CREATE TABLE LogEvent(tenantID INT,serverName VARCHAR(200), appName VARCHAR(200), priority VARCHAR(200),logTime DOUBLE,logger VARCHAR(800),message VARCHAR(3800))'); insert overwrite table Logs select tenantID, serverName, appName, priority, logTime, logger, message from LogEventInfo; select tenantID, serverName, appName, priority, logTime, logger, message from LogEventInfo;
18
![Page 19: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/19.jpg)
VisualizaXon -‐ Gadgets
19
![Page 20: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/20.jpg)
Demo ….. http://wso2.com/library/demonstrations/2014/02/screencast-monitoring-system-logs-with-wso2-business-activity-monitor/
20
![Page 21: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/21.jpg)
Use Case Scenario 2: HTTP Log Monitoring With WSO2 BAM
21
![Page 22: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/22.jpg)
HTTPD Logs and Use Cases • Monitor every web request information
– HTTP method – URI – Status code
• Monitor request properties – Message size – Host / IP address – Geo location – Date and time
22
![Page 23: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/23.jpg)
HTTPD Logs: Scenarios • Publish HTTP logs to BAM from a data agent • Split log entry for each fields and store in another big
data column family • Resolve IP addresses to geo locations • Aggregate request for geo locations and store into a
relational database (RDB) • Visualize from gadgets
23
![Page 24: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/24.jpg)
SoluXon Architecture
24
![Page 25: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/25.jpg)
Key Performance Indicator (KPI) Use Cases • Analysing request count variation for each host • Analysing request count variation for each hour • Analysing request message size variation
25
![Page 26: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/26.jpg)
Final Dashboard
26
![Page 27: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/27.jpg)
Demo ….. http://wso2.com/library/demonstrations/2014/02/screencast-http-log-monitoring-with-wso2-business-activity-monitor/
27
![Page 28: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/28.jpg)
28
Business Model
![Page 29: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/29.jpg)
29
Call to acXon page ๏ Be your own NSA: hhp://wso2.com/landing/nsa/
๏ Big Data Webinar Series: hhp://wso2.com/landing/wso2-‐bigbrother-‐webinar-‐series/
๏ WSO2 Business AcXvity Monitor: hhp://wso2.com/products/business-‐acXvity-‐monitor/
๏ WSO2 BAM Docs: hhp://docs.wso2.org/display/BAM240/WSO2+Business+AcXvity+Monitor+DocumentaXon
๏ WSO2 Complex Event Processor: hhp://wso2.com/products/complex-‐event-‐processor/
๏ WSO2 CEP Docs: hhp://docs.wso2.org/display/CEP300/WSO2+Complex+Event+Processor+DocumentaXon
![Page 30: NSA for Enterprises Log Analysis Use Cases](https://reader031.vdocument.in/reader031/viewer/2022020306/54c6a9074a7959b14b8b456b/html5/thumbnails/30.jpg)
Contact us !