next-generation itoa solution€¦ · of the use cases that itoa can be utilized for includes:...

INNOVATION PLATFORM WHITE PAPER 1

technicacorp.com

The Technica Innovation

Platform White Paper

Series presents

advanced topics that

will drive competitive

advantage for

next-generation IT

over the next

three-to-five years.

NEXT-GENERATION ITOA SOLUTION Next-generation IT Operations Analytics (ITOA) is the practice of using data

science principles, techniques, and algorithms on data that is collected from

underlying IT infrastructure, widening the lens through which to view available

data. This provides actionable insight into data for operational intelligence that

enables more effective decision-making and better service assurance. A sampling

of the use cases that ITOA can be utilized for includes: anomaly detection, pattern

discovery, correlation, and root cause analysis. These outcomes are simply a

matter of choosing the right data set to marry with the appropriate algorithm.

ITOA fuses Big Data concepts and technologies with traditional IT operations tools

to acquire, analyze, and visualize data which provides operators access to the

data they need for operational efficiency. ITOA provides insight into both

business execution and IT operations. To fully appreciate the value proposition

for Next-generation ITOA, it is helpful to understand the data center without

ITOA. We will contrast this as the “Tools-Driven” approach vs. the “Data-

Informed” paradigm present in Technica’s next-generation ITOA Platform.

Tools-Driven

In the effort to guarantee service level agreements and commitments—both

internally and externally—enterprises responded to IT “fires” by purchasing

products in an ad-hoc fashion based on vendor product or technology domain.

Figure 1 portrays examples of these various “point” solutions that perform a

particular job like monitoring applications, event management, network

performance, and security.

Each tool is implemented to focus on operating, administering, and maintaining

a specific IT domain. The individual tools can only function and respond to the

limited IT operations data set for which they have access.

This paper explains a next-generation

ITOA Platform that integrates three

vectors for effective IT operations –

Data-to-Data, People-to-Data, and

People-to-People. Technica’s

next-generation, ITOA Platform

ingests data from any data source

in the data center, persists the data

into a data lake, and then provides

analytics and configurable

visualizations, along with dynamic

and real-time collaboration.

Figure 1 – Tool-Driven Data Silos


technicacorp.com

Operators can only react to the information that an individual tool provides, and

must rely on manually interrogating and correlating data across an array of

individual tools. Each of these tools utilizes individual/proprietary access rights,

user interfaces, and nuances that inhibit agile responsiveness. This also degrades

the security posture of the enterprise due to lack of end-to-end visibility under a

single lens.

The complexities of a tool-driven approach magnify operational expenses with

continuous training on the individual tools due to infrequent usage and operator

churn, account management overhead, finger-pointing across administrative

domains that delay service restoration, duplication of effort with multiple

operations teams chasing the same problem, redundant tools, and use of

expensive subject matter experts to fight fires. With increased velocity of IT

solutions being implemented to deliver enterprise services, the net result of a

tool-based approach to IT Operations is siloed functions and fractured data within

the data center that promotes reactive instead of proactive operations and

inhibits service and security assurance for the enterprise.

Distributed data center locations add to the complexity of a tool-driven approach.

While disparate systems can be integrated to move towards a common operating

picture, this entails high capital and operational expenses. Additionally, the

end-solution typically lacks the agility required by dynamic business conditions

due to brittle data sharing interfaces that require tedious and lengthy integration

activities. Inevitably, one of the data formats changes and the entire cycle

must be repeated—often incurring system down-time until resolution.

Moreover, there is no overall service visibility—no overall ability to see services

enterprise-wide.

Data-Informed

Next-generation ITOA enables enterprises to become data-informed – fusing

human insight about the mission with unified data. The data-informed paradigm

enables the enterprise to move away from the islands of inoperable data and gain

visibility into aggregated and integrated data, from all the tools, making the data

accessible, useful, and valuable to operators and business managers.

Data generated by all the tools, devices, and sensors flow into a “data lake,”

where it can be viewed, interrogated, analyzed, and correlated in totality for

actionable insight. Conceptually this results in the situation pictured in Figure 2

(vs. Figure 1).

This data-informed approach takes advantage of analytics and visualizations to

democratize data, and provide a common operating picture for decision makers.

It enables the enterprise to extract meaningful information from the rapidly

growing landscape of disparate data sources and islands of applications.

The purpose of this paper is to explain a next-generation ITOA Platform that

integrates three vectors for effective IT operations – Data-to-Data, People-to-

Data, and People-to-People. Technica’s next-generation, ITOA Platform ingests

Figure 2 – A Unified Data Approach that Facilitates Overall Service Visibility


technicacorp.com

data from any data source in the data center, persists the data into a data lake,

and then provides analytics and configurable visualizations, along with dynamic

and real-time collaboration.

The Platform gives an organization an enterprise-wide operating picture of their

environment, using data analytics to provide better performance, availability, and

security analysis. At the same time, executives can make more informed

investments because they have an integrated end-to-end situational awareness

of the health and status of their enterprise. Thus, an enterprise can leverage

Technica’s ITOA Platform to improve efficiencies, agility, and responsiveness—

while at the same time reducing costs.

2. ITOA PLATFORM Over the years, Technica has developed numerous reference architectures,

including a Next-generation Big Data Reference Architecture and a Fog

Computing/Internet of Things (IoT) Reference Architecture. The research and

development of these reference architectures allows the company to investigate

capabilities before looking at individual components.

Then Technica moves to integrating actual components to create a prototype of

the reference architecture. During component selection, the company looks for

best-of-breed and cost-effective products. We define interfaces that ensure

vendor lock-in is prohibited. Additionally, many enterprises have already selected

products. In these cases, Technica uses the reference architecture approach to

extend the existing software assets to meet the required capabilities.

Technica’s ITOA Platform is based on its research and experience with data

analytics reference architectures. It can be visualized in a number of ways

including:

Functional View

The essential functions of the ITOA Platform are Collect, Analyze, and Act.

The Collect, Analyze, Act paradigm drives the functional view of the platform,

as shown in Figure 3.

COLLECT— The Collect function acquires, parses, and integrates data from

data sources. For complete visibility into IT operations at least four different

data sets should be captured:

Machine Data - Any digital data produced by devices, sensors, servers,

and applications, to include IT operations, administration, and

maintenance tools.

Wire Data - All Layer 2 to 7 communications between all systems.

Agent Data - Byte-code instrumentation and call-stack sampling. This

data has typically only been visible to the development and QA teams,

but is crucial for IT Operations moving toward continuous delivery and

the latest DevOps techniques.

Synthetic Data - From synthetic transactions and service checks.

Synthetic data originates outside the application delivery chain through

hosted monitoring solutions or as a part of active service checks, e.g.,

ICMP pings to fully scripted checks, etc.

Figure 3 – Functional View of ITOA Platform


technicacorp.com

ANALYZE— This function includes configurable visualizations, the data lake,

and specific analytics including search and more sophisticated algorithms.

Examples of Technica-developed algorithms to extend off-the-shelf

capabilities include anomaly detection and tangential data references.

ACT— These are the outcomes based on the data being appropriately

collected and analyzed. Depending on the use case, alarms can be generated,

trouble tickets can be created, etc. Additionally, collaboration and problem-

solving might be needed to derive a resolution.

Component View

Technica integrated the following components into the reference architecture

(Figure 4) to meet the Collect, Analyze, Act mantra.

Figure 4 – Component View Reference Architecture

Technica elected to use Logstash for Acquisition and Integration of IT operations

data. It collects, parses, and transforms the operations data. Then, Logstash

persists the data to Elasticsearch for Analytics. Elasticsearch serves as the data

lake and provides robust and scalable search capabilities. Finally, Technica

selected SitScape to provide visualization and high-level analytics to end-users.

Importantly, SitScape has extremely robust collaboration features that leverage

enterprise-grade security to enable best-of-breed User Defined Operational

Pictures (UDOPs).

Other products can be utilized—each with different strengths and weaknesses.

For example, Apache Kafka can be used as an alternative to Logstash. If streaming

analytics were desired, in addition to the batch analytics available via the

Elasticsearch data lake, Kafka could play an important part in enabling a Lambda

design pattern within the ITOA Platform. Hadoop or another no-SQL database,

like MongoDB, can be selected for the data lake.

However, for most use cases, Technica believes the aforementioned components

provide the best value.


technicacorp.com

The Technica ITOA Platform provides full-service visibility across the enterprise

incorporating the following objectives:

Data consolidation into a data lake that serves as a single integration point

Enhanced decision-making aided by both high-level and deep analytics

Increased operational efficiencies through self-service,

zero-programming access to data

Better collaboration between operations support staff

User-defined Visualizations

Open Application Programming Interfaces (APIs) minimize vendor lock-in

and ensure interfaces are not brittle

Infinitely scalable architecture, using cloud-enabled technologies on

commodity infrastructure

“Zero-programming” is enabled by SitScape’s intuitive interface for integrating

new data sources, performing analytics, and creating UDOPs, shown in Figure 5.

Figure 5 – SitScape Visual Editor for Data Analytics Processing

The SitScape Visual Editor couples a flexible data analytics process design with a

drag-and-drop user interface. This interface enables optimal data management

and operations, while maximizing flexibility and usability.

Scaling the solution for the cloud allows capacity to be added on-demand, only

when needed. This valuable feature is enabled by the products selected for the

ITOA Platform, combined with Technica’s vast experience with cloud-scaling

technologies. Cloud performance and operational availability are assisted with

in-memory cache, distributed cloud-enabling infrastructure techniques and

software, and replication/high-availability mechanisms.


technicacorp.com

3. ITOA PLATFORM CORE CAPABILITIES AND BENEFITS The ITOA Platform provides a synergistic and adaptable architecture that meets a

variety of ITOA use cases.

Figure 6 portrays the elements: software-defined computing, collaboration,

analytics and visualization, and correlation synergistically providing a securely

shared data management framework for effective decision making. Open APIs

and cloud-enabling technologies/concepts ensure quick and effective preparation

and transformation of data from disparate sources for analytics.

The core capabilities function harmoniously to provide a system that is secure,

robust, and reliable—with multiple levels of access controls. Additionally, the

ITOA Platform enables agile development and fielding of modular capabilities.

The Platform’s architecture enables IT operations to become a strategic

advantage, providing an integrated, near real-time Common Operational Picture

(COP) of the IT landscape necessary for advanced collaborative data-informed

decision making.

Figure 6 – A Collaborative Data-Driven Framework for Effective Decision Making

Collaboration

With zero-programming, IT operators quickly create UDOPs that can be shared,

allowing advanced collaboration and optimized decision actions through rich and

dynamic messaging capabilities. Advanced role-based or attribute-based security

architectures ensure that only the right people get access to authorized

information.

Rapid Visualization and High-Level Analytics/Correlation

The user selects the appropriate data sets, tracks key metrics, and analyzes

salient trends to quickly create UDOP visualizations. If needed, actions can be

triggered based on user-defined criteria. Thus, end-users are enabled to create

user-defined, dynamic, web-based visualizations of big data sets using an array of

data sources, as exemplified in Figure 7.


technicacorp.com

Figure 7 – Example UDOP

The user-defined functionality extends to high-level analytics, deep analytics,

data correlation, numerous drill-down visualizations, and dashboard options—all

without software programming knowledge or skills. The dashboard options

include data-driven, real-time monitoring and alerting.

Technica’s ITOA Platform easily integrates data from any system to ensure

capture of all available data that may be relevant to the decision-making process.

The data acquisition and analytics engine discovers connections and patterns that

are not apparent with traditional approaches.

Often, given the number of point-solutions as pictured in Figure 1, enterprises are

not even aware of all data that is available. In other words, the ITOA Platform’s

data Collect function surfaces data that enterprises may currently be overlooking.

Moreover, the Collect function allows the enterprise to combine information in

ways not possible before—thereby revealing new insights.

The flexible visualization components allow users to realize the full value of

the system features without in-depth knowledge of the underlying technology.

The customizable dashboards present the information visually and intuitively,

allowing users to focus on the data and information—not the technology.

Synergistic Agility

Current tools-driven IT operations systems lack the flexibility and scalability to

incorporate agile development processes and DevOps workflows, especially as

enterprises incorporate containerization and microservices architectures.

Technica’s ITOA Platform delivers an agile solution that increases the velocity

of data acquisition, integration, management, and visualization.


technicacorp.com

Self-service software-defined technologies are built on modular off-the-shelf

components that utilize an open architecture for interoperability with web-scale

features.

A key enabler for the adaptability and scalability of the ITOA Platform is the

underlying modular and component-based approach with an intent-based

analytic framework, and its no-programming required design principle.

These architecture and design tenets enable enterprises to rapidly develop new

solutions for incremental capabilities with continuous feedback from prototype

and testing through the deployment and sustainment activities.

Data collection, categorization, aggregation, tabulation, and analytic processing

for analysis and decision making is assured through a robust data pipeline

assurance system that manages the integration of the modular components for

data acquisition, fusion, inspection, analysis, and visualization of data from

disparate and federated sources.

The data Collect function of the ITOA Platform will auto-ingest or manually ingest

data, and provide built-in storage and caching for ingested data, as well as use

external databases, Big Data stores, sensors, and legacy applications and data

sources. The formats include but are not limited to relational, CSV, Excel, log files,

Netflow data, JSON, unformatted text, Microsoft Word, PDF, static web sites,

image, and video. This built-in adaptability allows the enterprise to meet new

requirements without vendor support services.

Other Critical Platform Capabilities

Technica’s ITOA Platform incorporates the following additional capabilities:

DATA SYNCHRONIZATION— Supports the acquisition and ingestion of data

from multiple concurrent and federated sources. It includes capabilities that

ensure guaranteed delivery of data; integration with Disrupted,

Disconnected, Intermittent, and Limited (D-DIL) bandwidth environments

that load data only periodically or may go down for extended periods of

time; and supports backward capability with legacy applications.

DATA MANAGEMENT— Performs scheduled and ad-hoc Extract, Transform,

Load (ETL) functions, query, discovery and visualization of structured,

unstructured and semi-structured data. It supports a flexible and holistic data

architecture that addresses data in motion, data stores, data items and

mapping of data artifacts.

SMART DATA DISCOVERY— Supports smart data discovery through its

machine learning capability that automatically models the behavior of

ingested data, and identifies trends, periodicity, and issues in real time to

streamline root cause analysis. This increases the effectiveness of data

management and reduces the burden of configuring and manipulating tools

by automatically learning a predictive model for the distribution of feature

values at a given time, based on the historical values seen to date. This

predictive model computes the probability of current behavior given

historical behavior.

RESILIENCY— Supports high availability by design. The data architecture is

based on sharding, providing 100% data availability.


technicacorp.com

DYNAMIC DATA MODEL— Supports the automatic detection of data field

types paired with the ability to dynamically add fields to a given index.

A user could start saving new information with each record without having

to change or define new schemas. This flexibility provides a means to

get started quickly and to iteratively improve the schema as needs

become clearer.

INTEROPERABILITY— Provides a REST API for integration with external

systems and applications, with low-level ingest plugins that support TCP

and UDP messages allowing for ease of ingest for most network protocols

that other tools produce and consume.

ROBUST SECURITY— Provides multiple layers of security with robust access

controls to compartmentalize data. The layered security provides protections

for data in transit and data at rest, down to the data field level. The system

controls who can add or delete a document in an index, who can access

sensitive documents, and who can access individual fields in documents.

The auditing feature maintains a complete record of all system and user

activity, with an alerting feature that can be used to proactively notify

appropriate personnel of changes or approaching thresholds which are

relevant to security compliance.

4. TECHNICA’S VALUE ADD Technica maximizes the use of commercially available solutions and integrates

them to deliver agile and effective capabilities to our clients. Over time, Technica

has developed value-add capabilities in the areas of Analytics, Artificial

Intelligence (AI), and Cloud Computing Infrastructure & Orchestration that can be

integrated into the ITOA Platform to meet mission-specific unique requirements.

Broad Experience/Past Performance

As a company that has been around for more than 25 years, Technica possesses

deep experience in the Federal, DoD, and Intelligence Community spaces.

Moreover, Technica possesses the requisite skill sets needed to implement the

ITOA Platform, including extensive experience with integration, cloud scaling,

Big Data, and ETL.

Technica IR&D

Additionally, the company maintains a division devoted to Independent Research

and Development (IR&D). This is exceedingly rare for a company the size of

Technica. IR&D focusses on next-generation technologies, especially AI

algorithms that utilize machine learning, graph analytics, and deep learning.

IR&D’s algorithms and solutions directly apply to the ITOA Platform for certain

use cases.

FLASHLIGHT— Flashlight is a plug-in for Elasticsearch developed by

IR&D. Flashlight provides edge/vertex graph analytic capabilities (path,

connectivity, community, and centrality analysis) and machine learning to

connect “dark data” or fringe data associated with a query result, and

performs predictive analytics. Flashlight can be used to enable

user-defined graphs, reports, charts, tables, geospatial maps and other

desired visualizations for IT operations situational awareness.


technicacorp.com

AI ANALYTICS— IR&D has devoted a great deal of time developing

next-generation algorithms that can be used to enhance the high-level

analytics present in the ITOA Platform. Most of these AI analytics leverage

deep learning to train models for prediction. The algorithms are packaged

as Docker-based microservices. The AI Analytic Microservices Catalog

available to the ITOA Platform are:

Deep Learning Algorithms

o Fall Detection—Convolutional Neural Network (CNN) o Image Classification—CNN o Time Series Analysis—Recurrent Neural Network (RNN) o Long Short-Term Memory (LSTM) o Anomaly Detection—Autoencoder

Genetic Algorithms to Hyper-tune Parameters

Federated Learning

A complete discussion of these AI Analytic Microservices and deep learning is

beyond the scope of this document, but the following Anomaly Detection section

details a specific use case of anomaly detection for ITOA.

5. ITOA USE CASES ITOA use cases are as endless as IT Operations data streams/applications.

This document presents two current use cases Technica is advancing. This list

will grow over time.

Vulnerability Management

Tracking and remediating vulnerabilities, such as unpatched software and

operating systems and the installation of unsupported applications, plays a

crucial role in ensuring availability and integrity of an enterprise’s services.

Technica’s ITOA Platform can be utilized to strengthen an enterprise’s

vulnerability management systems by correlating the data from disparate tools

that manage security auditing, patching, asset inventory, and software inventory,

bringing to light new, actionable insight.

Figure 8 – Screenshot and Architecture of Vulnerability Use Case using SitScape

SECURITY SERVICES

DATA PIPELINE ASSURANCE

BrowserClient

MobileClient

RichApplication Client

ACQUISITION INTEGRATION ANALYTICS OPERATE VISUALIZE

SCCM

Data Sources

SCOM

ACAS

HBSS

Tanium


technicacorp.com

For example, in one real-world data set, the correlation of security auditing and

asset inventory data revealed that the security auditing tool was scanning certain

assets multiple times and inflating the vulnerability numbers every week.

Figure 8 portrays a screenshot and architecture of an actual ITOA vulnerability

implementation.

Beyond data correlation, trending analysis is necessary to isolate issues in

vulnerability management processes and to construct predictive scenarios.

The Architecture provides the capability to automate this process. This, in turn,

lowers the mean time to identify and resolve critical issues, such as the

stagnation of a number of hosts afflicted with a vulnerability due to a halt in

patching. This also frees up analysts and subject matter experts to proactively

pursue innovations, rather than reactively responding to small problems, i.e.,

“firefighting.”

Anomaly Detection

Technica IR&D developed an Artificial Neural Network (ANN) deep learning

algorithm based on an Autoencoder—the Anomaly Detection Microservice.

As portrayed in Figure 9, an Autoencoder is a dense neural network with a

decreasing number of nodes per layer until the middle.

Figure 9 – Notional Autoencoder ANN Architecture

The second portion of the ANN architecture is a mirror image of the first half. As

the Anomaly Detection Microservice is trained, the ANN works to recreate data

streams. As it sees more of the same data, it gets better at recreating the data.

However, when an outlier data stream is presented to the model, the Anomaly

Detection Microservice has trouble recreating the stream. This is an anomaly.

When an anomaly is recognized, a configurable array of alarms, alerts, and

notifications can be generated.

Technica’s AI Anomaly Detection Microservice can be applied to any data

captured in the Collect function. Thus, anomalies could be spotted in vulnerability

management, DevOps, Application Performance Monitoring (APM), network

performance, capacity management, etc.

Inp

ut

Ou

tpu

t

Compressed Feature Vector

DecoderEncoder


technicacorp.com

CONCLUSION There is no one-size-fits-all approach to ITOA. Technica’s ITOA Platform is

modular and component-based. The Platform delivers the right information, to

the right people, at the right time. This capability ensures effective decision

making by providing a comprehensive, scalable, end-to-end data management

and operations capability with an architecture that is highly adaptable and

extensible to meet the dynamic and evolving data requirements of the data

center.

Pervasive throughout the ITOA Platform are touchpoints for Technica value-add

services and innovative capabilities that enable and facilitate effective, agile data

management and IT operations.

Technica’s ITOA Platform enables the required agility for IT operations by

minimizing redevelopment and customization activities—focusing on analytics,

functionality, and visualization through graphical-based configuration and

integration utilities.

Technica provides professional

services, products, and innovative

technology solutions to the Federal

Government. We specialize in network

operations and infrastructure; cyber

defense and security; government

application integration; systems

engineering and training; and product

research, deployment planning,

and support.

22970 Indian Creek Drive, Suite 500

Dulles, VA 20166

703.662.2000

©2019 Technica Corporation. All rights reserved.

next-generation itoa solution€¦ · of the use cases that itoa can be utilized for includes:...

Documents