big data visualization problem in it management
DESCRIPTION
Big Data Visualization Problem in IT Management Serge Mankovski, Research Staff member at CA Technologies January 22nd 2014 Big Data exposes and amplifies need for understanding big picture of what data tells you without loosing track of minute details. This need manifests in different aspects of Big Data from data processing and management to visual analytics and insight. This problem clearly manifests in management of very large IT infrastructures. CA Labs, as a part of CA Technologies, is focusing on the problem of visualization of Big Data in context of traditional IT management problems of root cause analysis, impact analysis, and change management. We will demonstrate examples of the Big Data Visualization problem as we see it and try to somehow outline this area of research problems. Big Data Visualization Meetup - South Bay http://www.meetup.com/Big-Data-Visualisation-South-BayTRANSCRIPT
![Page 1: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/1.jpg)
Serge Mankovski CA Labs Research Staff Members
Big Data Visualization in IT Management Environment
![Page 2: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/2.jpg)
Problem as we see it
— Data is more complex than ever before − Three Vs of Big Data − Mushup of structured, semi structured, and unstructured − Longer time frames
— and we are eager to use long term data more than ever before − Big promise of Big Data
— Existing visualization techniques mostly built for less complex data
— Time constraints for interactive visualization remain as before
— It is clear in IT management that new techniques are needed to accomplish common use cases
![Page 3: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/3.jpg)
Problematic Use Cases In IT MGMT
Elements of interest − Single element • Server
• Router
• Database
− Groups • Service • Network • Location
Relationship • Server A hosts Virtual Machine B
• Application A uses Database B
• Service A is contains Application B
3
− Datacenter • Multiple datacenters @ 20,000+ servers each with
several hundred items
− Mainframe • Single LPAR can have 30,000+ items
− Cloud Connected Enterprise • All of the above plus partially transparent cloud
deployments
![Page 4: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/4.jpg)
4
This problem exists in many places
![Page 5: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/5.jpg)
The Dilemma
5
Access to detail
Reduce screen complexity
![Page 6: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/6.jpg)
6
Our approach to resolving the dilemma
— Common Approach − Show as much as possible − Use various layouts − Use overview and zooming
— Our Approach − Show as little as possible − Use simple layout − Use semantic zooming and
layered overview
![Page 7: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/7.jpg)
Map of IT Environment
— A map of the user’s workspace, where elements of the IT environment are assigned to a layered structure that allows the user to quickly recognize dependencies between areas of the network
7
Enterprise Asset Terrain
Layer 1
Layer 2
Dependency
![Page 8: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/8.jpg)
Definition of Layers
We define a layer in terms of:
− A set of elements (or aggregations) we want
to visualize on the layer
− The resources these elements share
8
![Page 9: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/9.jpg)
Defining Hierarchical Layers Step 1 – Assign elements to the layer
1. Select subset of elements we want to visualize
2. Assign elements to layer
Services layer
9
![Page 10: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/10.jpg)
Layers Definition IT Environment elements mapped to layers
Abs
trac
tion
Services Applications Systems Networks Assets
10
![Page 11: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/11.jpg)
Defining Hierarchical Layers Step 2 – Select related resources
1. Select type of resource dependencies to visualize
2. Select relevant relations between layer element and related resources
Services layer
11
![Page 12: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/12.jpg)
Defining Hierarchical Layers Step 3 – Detect and draw dependencies
1. Identify layer elements with shared resources
2. Draw layer elements to represent overlapping of resources
Services layer
12
![Page 13: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/13.jpg)
Visualization Map Creation Elements overlap for each level
Abs
trac
tion
Services Applications Systems Networks Assets
13
![Page 14: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/14.jpg)
A Map of the IT Environment Semantic navigation of complex environment
Services Applications Systems
Assets Networks
14
![Page 15: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/15.jpg)
Complete Map of the IT Environment Visual Summary of the Environment based on Filters
Service 5 is selected
Services
Related items in all other levels are highlighted
Applications Systems
Assets
15
![Page 16: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/16.jpg)
Navigation
Service 5 Service 5
OR Service 4
Service 5 AND
Service 3 16
![Page 17: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/17.jpg)
Getting to What is Important
Start with a complex IT environment
Use simple filters to create an abstract representation of IT
infrastructure
Open areas of interest in context using traditional
visualization tools
17
![Page 18: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/18.jpg)
Semantic Aspects of Layer Formation
18 Innovation by CA Labs Copyright © 2010 CA. All rights reserved. February 21st 2012
![Page 19: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/19.jpg)
Original System
19
![Page 20: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/20.jpg)
Select Green Elements
Select Blue Elements
Defining Blue-Green Layer by Type of Element
20
Select Red Relationship
![Page 21: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/21.jpg)
Naming Surfaces of the Layer
21
APPLICATION
SERVCE
USES
Top Class
Bottom Class
Relationship Set
![Page 22: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/22.jpg)
Equivalence
22 CEWIT 2011 November 3rd, 2011
Equivalency in Top and Bottom Classes
red = gray yellow = green
Equivalency in Relationship Set (yellow = green)
![Page 23: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/23.jpg)
23 CEWIT 2011 November 3rd, 2011
Transitive Closure
3
5 6 4 7
1 2
9 8 10
3
5 4
1 2
9 8 10
1 -> 4 -> 8 => 1 -> 8 2 -> 4 -> 8 => 2 -> 8 2 -> 5 -> 9 => 2 -> 9 3 -> 5 -> 9 => 3 -> 9 3 -> 6 -> 8 => 3 -> 8 3 -> 7 -> 9 => 3 -> 9
Becomes
![Page 24: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/24.jpg)
Storage behind visualization
24 Innovation by CA Labs Copyright © 2010 CA. All rights reserved. February 21st 2012
![Page 25: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/25.jpg)
Graph Cache Behind the Visualization
Graphic Renderer
Web Service
Database APP A
PP
APP
DB DB
DB DB
Graph database querying system
3rd party application
25
![Page 26: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/26.jpg)
Graph cache performance gains
26
![Page 27: Big Data Visualization Problem in IT Management](https://reader034.vdocument.in/reader034/viewer/2022051323/549596a8b479591b158b4614/html5/thumbnails/27.jpg)
27
Questions?