presented by

Unstructured Data To Interactive Graph

Jacob KomarovskiXpoLog, Turn Data Into Action

presented by

XpoLog – End to end log management solution• Collection of logs data from different sources • Automated analytics engine utilizing NLP• Visualize logs data using dashboards and widgets• Continues monitoring and alerting• Enterprise ready

XpoLog standard cluster• Collects 1.5+ TB of data per day• Analyze 1000’s of different logs

XpoLog – Log Management

presented by

• DevOps and Operations– Log Analysis Platform

– Search events and patterns

– Troubleshoot and Analyze IT and Apps

– Reporting and Monitoring

• IT Data– ML and NLP Powered Search– Identify bottlenecks and faults– Analyze Audit Data– Quickly identify risk

XpoLog – Use Cases

presented by

Why visualize log data as a graph?• Easy exploration

– Show the big picture– Visualize cross logs transactions– Fast root cause analysis

• Data Augmentation – Paths coloring– Easy visualization of enhanced data

• Rules• Tags• AI\ML engines• Risk

Why Graph

presented by

Explored and tested several technologies• Main feature is UX

– Performance – in memory, uses sparse adjacency matrices– Attributes on Nodes and Edges – Labeled Nodes – Consolidate different types of data sets

• On Premise solution – Easy deployment– Industry standard– Easy to grow with the data - Clustering

• First implementation of 3rd party graph solution– Because of YOU – great community

Why Redis Graph

presented by

XPLG Data Flow

Data Collection

Connectors• Push• Pull• Cloud

Automated Pattern recognition

Running Processing Engines

• Index

• Analytics

• Rules

• Graph

UI Visualization

• Search console

• Dashboards

• Graph

Automation

• Complex Algorithms• AI and NLP• Graph theory

applications

• Monitor & Act • Rest API calls• Scripts

• Alert• Slack, Teams…• Mail

presented by

Graph Data Flow

Log4J

ParserNLP and Statistical Analysis

XplgGraph

Redis Graph

UI Model

D3 Visualization

Monitoring & Alerting

Data Augmentation

Configuration FileStructured DataRule Engine

presented by

Add ability to create graph model on any type of log in different contexts• XpoLog Parsers results in structed data• Graph structure defined using a configuration file

– Nodes structure and representation in the DB– Connections between nodes– UI representation attributes

Generic Graph Parser and Engine

presented by

Graph Structure• Every real world entity (e.g. user, server….) is

represented by one node in the graph• Every real world transaction is represented by

one edge in the graph

Our use case• Initially, mostly new nodes• After a while, small number of new nodes and

mostly new edges

Adding New Data To The Graph

presented by

Merging a small graph into a large graph• Merge every node• Create all edges

Observations• Nodes should be cached (using Redis for example)• After a while, almost no nodes will need to be

merged• Graph merging is challenging …

Adding New Data To The Graph

presented by

Query• Is a Graph object• Same as xplgGraph• Every node and edge have a filter that we apply on the graph

Response• Is also an xplgGraph• Has the same structure as the query graph

Transforming Java object to Cypher query

presented by

Java to Cypher• An AST is built from the query graph and then transformed into a Cypher query

compatible with RedisGraph• Achieved by building on top of jCypher (https://github.com/Wolfgang-

Schuetzelhofer/jcypher)• JRedisGraph (https://github.com/RedisGraph/JRedisGraph) is used to

communicate to Redis

Transforming Java object to Cypher query

presented by

Data visualization• Bi directional data flow

– Graph model translated to json structure with UI properties– User interactions translated to a query graph and then transformed into a Cypher query

compatible with Redis Graph

• Visualize using D3 library– Wrap D3 inside XpoLog product

Graph Model -> UX

presented by

Goal - combine graph representation in XpoLog Dashboards, running over 1TB data per day for up to 5 years• Split stored graph – creating a forest• Create pre-processed aggregated graphs

– Aggregate overtime in advance

• Using dumps for hot/cold data

Future Challenges - Dealing With Big Data and UX

presented by

Running continues testing from scripts and XpoLog application• First graph building from existing environment

– 10#$ of events from different logs normalized to graph structure• Insert in bulks• Side process• Hot/Cold storage

– Many new nodes and edges

• Continues graph maintenance – Simulate data collection process

• Non stop update of nodes• Adding new edges

– Maintain pre-processed aggregations

Testing Benchmarks

presented by

DEMO

presented by

Thank you!