ensign on hpe superdome flex ensign on hpe ......ensign® on hpe superdome flex how in-memory...
TRANSCRIPT
ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic
ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic
ENSIGN ENSIGN® is a scalable high-performance data analytics software package developed by Reservoir Labs that decomposes large datasets (for example, network logs) into recognizable patterns of behavior and provides actionable insights into the data. The core of ENSIGN is a suite of tensor decomposition routines that are optimized for both shared memory and distributed memory systems. ENSIGN has successfully extracted normal and off-normal patterns, leading to the discovery of anomalous and alarming behaviors, from logs with billions of entries.
URL: www.reservoir.com/ensign-cyber/
ENSIGN Features and Benefits
Feature Benefit
No up-front specification of what to look for
Enables the user to detect unknown unknowns
Easy-to-use command-line tools to process, ingest, and export network data
Simplifies data ingest/export and avoids complex feature engineering on input data
Scriptable and modular tools for running the workflow
Enables automated workflow and easier integration into existing workflow/platform
Integration with modern ML packages and workflows
Integrates with Python tools and environments such as Anaconda and Jupyter notebook – makes ENSIGN usable by a data analyst along with other tools
Joint analysis of multiple data logs
Enables comprehensive and deeper analysis and insights
Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic
ENSIGN Features and Benefits (continued)
Feature Benefit
Patented and patent-pending high-performance optimizations for core analysis methods
Makes ENSIGN tensor decompositions applicable for large-scale cyber data analysis (level of capability and scalability is well beyond what open source tensor tools could offer)
Advanced patent-pending data structures (“mode specific sparse tensor” and “mode generic sparse tensor”) for tensor methods
Enables effective utilization of compute and memory resources in target compute systems and thereby enables the application of advanced tensor methods for extracting complex patterns
HPC deployments [Large shared or distributed memory systems. Local or in the cloud.]
Leverages 100s or 1000s of cores in the system with maximum utilization and minimum data footprint – enables scalable large-scale deployment
Streaming (“low-rank”) updates Helps the user understand, in near-real-time, how the network behavior is evolving over time
Richer set of advanced analysis methods
Enables identification and interpretation of sophisticated network behaviors
Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic
ENSIGN Requirements
Operating Environment
Linux
Hardware Platform
Desktop PC 100,000 scale*
Multi-core Server (20 cores, 500 GB) 10-100 million scale
High-end In-memory Server (896 cores, 48 TB) 100 million - 100 billion scale
Cluster of Multi-core Nodes 100 million - 10 billion scale
Human Resources
No tensor expertise, familiarity with data
Software Components
Primary software: ENSIGN v4.2
Dependent software packages: a. Python (v3.7.0) b. Dask (v1.0.0) c. numpy (v1.15) d. matplotlib (v3.0.3)
*scale: indicates the number of log entries in the data file
Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic
Superdome Flex Superdome Flex is a uniquely modular, highly flexible and reliable platform that scales seamlessly to help businesses of any size turn critical data into real-time business insights.
Superdome Flex Features and Benefits Turns data into actionable insights at unparalleled scale and in real-time
• HPE Superdome Flex Server offers unparalleled scalability of up to 32 sockets • In-memory design and unmatched memory capacity of up to 48 TB in a single
platform
Superdome Flex Full rack
(Hero shot)
Superdome Flex (Rear 16-slot)
SuperdomeFlex(Topinternal)
Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic
• Gives the compute power needed for most demanding in-memory workloads with groundbreaking performance at scale, ultra-low latency, and high bandwidth.
• Designed for the future based on Memory-Driven Computing design principles to boost analytics performance
Enables to keep pace with evolving in-memory computing demands
• HPE Superdome Flex Server has a unique modular design that scales flexibly and seamlessly from 4- to 32-sockets in 4-socket increments; utilizes Intel Xeon Scalable processors both Gold and Platinum.
• Offers cost efficient entry point for mission-critical workloads at 4 sockets and the ability to scale beyond 8 sockets with both cost-efficient Gold processors and premium Platinum processor
• Allows to scale up or scale out the environment with 4-socket modular building blocks and customize each building block to match workload needs with choice of memory size and capacity, processor and core count, and amount and type of I/O
Safeguards mission-critical workloads
• The HPE Superdome Flex Server delivers the highest service levels on industry standards with extreme and proven RAS capabilities not available on other x86 platforms
• Reduces human error with best-in-class predictive fault handling Error Analysis Engine, which predicts hardware faults and initiates self-repair without operator assistance
• Contains errors at the firmware level, including memory errors, before any interruption can occur at the Operating System layer with HPE's “Firmware First” approach
• Allows to isolate workloads and/or consolidate multiple workloads onto a single managed complex with Hewlett Packard Enterprise unique x86 hard partitioning (HPE nPars) and service individual partitions and/or reconfigure while other partitions continue to run undisturbed
• Delivers continuity for Linux® workloads with HPE Serviceguard for Linux (SGLX) high availability and disaster recovery clustering solution, protecting from a multitude of infrastructure and application faults across physical or virtual environments over any distance
Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic
ENSIGN on Superdome Flex ENSIGN and Superdome Flex offer a unique software-hardware solution to turn massive amounts of data into critical actionable insights. In particular, “ENSIGN on Superdome Flex” has been demonstrated in the cybersecurity space to accelerate data analytics, tackle large-scale workloads, and enable critical applications. ENSIGN fully and seamlessly utilizes the compute power offered by Superdome Flex for demanding in-memory workloads and enables quick turnaround time for cyber analytics on massive rapidly growing data. ENSIGN enables high performance at scale effectively exploiting the ultra-low latency and high bandwidth benefits offered by Superdome Flex for high-end in-memory computing.
ENSIGN-Superdome Flex performance on analyzing a large cyber log from a real operational network – showing near-ideal scaling of computation time, good scaling of communication time, and good scaling of
overall execution time of analysis
Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic
The highlights from the experiments of running ENSIGN on a “16 socket 288 core 12 TB” Superdome Flex box (as illustrated in the figure above) for analyzing a large cyber dataset collected from a real operational network are as follows:
• Able to run large workloads that was not previously possible (for example, workloads not possible with running ENSIGN on a commodity scale-out cluster of Intel Xeon nodes)
• Reduced the end-to-end cyber analysis workflow time from 5 hours down to 5 minutes for a ~10 GB cyber log collected over one day
The accelerated ENSIGN analytic solution is made possible by key optimizations that exploit the distinctive features of Superdome Flex.
Superdome Flex Feature ENSIGN Optimization
Low latency & high bandwidth memory performance
Reduced communication volume and synchronization in ENSIGN methods
Large scaled-up in-memory capacity Memory-efficient parallel computations in ENSIGN methods
Avoiding writing data to disk and keeping data in memory through the ENSIGN workflow
Scalable processing power Operation-minimal parallel computations in ENSIGN methods enabling near-ideal scaling of computations
(212) 780-0527 www.reservoir.com/ensign-cyber/ [email protected]