dell emc isilon onefs: apache hive low latency …...apache hive 3 with llap 7 dell emc isilon...
Post on 02-Apr-2020
20 Views
Preview:
TRANSCRIPT
H17672
Technical White Paper
Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance Running sub-second SQL on Apache Hadoop with Dell EMC Isilon F800 all-flash scale-out NAS storage and Isilon OneFS 8.1.2
Abstract This paper describes the support and performance test results for running
Apache® Hive™ 3 low-latency analytical processing (LLAP) with Dell EMC™
Isilon™ F800 all-flash NAS storage for sub-second queries.
April 2019
Revisions
2 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
Revisions
Date Description
April 2019 Initial release
Acknowledgements
Author: Kirankumar Bhusanurmath (Kirankumar.bhusanurmath@dell.com), Analytics Solution Architect
The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any software described in this publication requires an applicable software license.
Copyright © 2019 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its
subsidiaries. Other trademarks may be trademarks of their respective owners. [4/9/2019] [Technical White Paper] [H17672]
Table of contents
3 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
Table of contents
Revisions............................................................................................................................................................................. 2
Acknowledgements ............................................................................................................................................................. 2
Table of contents ................................................................................................................................................................ 3
Executive summary ............................................................................................................................................................. 5
1 Apache Hive 3 with LLAP ............................................................................................................................................. 6
1.1 The importance of fast queries in big data ......................................................................................................... 6
1.2 Why LLAP is used .............................................................................................................................................. 6
1.3 How LLAP makes queries easier ....................................................................................................................... 7
1.4 Apache Hive with LLAP architecture overview ................................................................................................... 7
1.5 Optimizations ...................................................................................................................................................... 8
1.6 Benefits of LLAP ................................................................................................................................................. 9
2 Dell EMC Isilon F800 all-flash NAS ............................................................................................................................ 10
2.1 Isilon nodes ....................................................................................................................................................... 10
2.2 Network ............................................................................................................................................................. 11
2.2.1 Back-end (internal) network .............................................................................................................................. 11
2.2.2 Front-end (external) network ............................................................................................................................ 11
2.3 File system structure ........................................................................................................................................ 11
2.4 Why use Dell EMC Isilon for big data and analytics ......................................................................................... 11
3 Solution overview ....................................................................................................................................................... 13
3.1 Isilon for big data analytics using Apache Hive ................................................................................................ 13
3.2 Hortonworks HDP 3.0.1 with Apache Hive 3 and Isilon topology ..................................................................... 14
4 Tested configuration ................................................................................................................................................... 15
4.1 Hadoop cluster.................................................................................................................................................. 15
4.2 Compute nodes ................................................................................................................................................ 15
4.3 Isilon F800: presented as NameNode/DataNode to Hadoop cluster ............................................................... 15
4.4 Hive testbench .................................................................................................................................................. 15
5 TPC-DS ANSI SQL compliance and performance test results .................................................................................. 16
5.1 HDP 3.0.1 configuration ................................................................................................................................... 16
5.2 Hive LLAP SQL compliance test ...................................................................................................................... 16
5.3 Hive LLAP SQL performance test .................................................................................................................... 17
6 Conclusion .................................................................................................................................................................. 20
A Configuration details ................................................................................................................................................... 21
A.1 Hadoop cluster configurations .......................................................................................................................... 21
A.1.1 OneFS service configurations .......................................................................................................................... 22
Table of contents
4 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
A.1.2 YARN service configurations ............................................................................................................................ 23
A.1.3 Hive service configurations ............................................................................................................................... 25
A.1.4 YARN queue manager ..................................................................................................................................... 29
A.2 Isilon configuration ............................................................................................................................................ 29
A.2.1 OneFS HDFS settings ...................................................................................................................................... 32
A.2.2 OneFS TCP tuning ........................................................................................................................................... 33
B Technical support and resources ............................................................................................................................... 34
B.1 Related resources............................................................................................................................................. 34
Executive summary
5 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
Executive summary
The most significant feature introduced in Apache® Hive 2 was LLAP (low-latency analytical processing).
LLAP enables sub-second SQL analytics on Apache Hadoop® by intelligently caching data in memory with
persistent servers that instantly process SQL queries. Since LLAP is an evolution of the Hive architecture, it
does all of this with the same comprehensive ANSI-standard SQL support and proven scale that Hive is
known for, enabling Hive to run all 99 TPC-DS queries with only trivial modifications to the original source
queries.
One of the most exciting new features of Dell EMC™ Isilon™ OneFS™ 8.1.2 from is the official support for
Hortonworks HDP 3.0.1 with Apache Hive 3 LLAP. This paper describes the official support and performance
test results (Apache Hive LLAP SQL compliant) for a Hortonworks HDP 3.0.1 cluster on generic servers for
compute and Isilon F800 for storage.
This paper also shows how the Isilon F800 scale-out NAS solution provides compliance to ANSI-standard
SQL for LLAP, excellent performance, and better storage utilization with less drives and a smaller storage foot
print than legacy solutions.
Apache Hive 3 with LLAP
6 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
1 Apache Hive 3 with LLAP One common use case in enterprise computing is the repeated use of the same tables. Business queries tend
to focus on operations like aggregation and on extracting a small amount of data out of a large data set. In
recent years, most Hadoop customers use a rudimentary approach of moving data from a Hadoop cluster to a
relational database for interactive querying. This introduces a large latency as well as additional cost for
management, and makes maintenance of multiple analytic solutions difficult. LLAP allows data scientists to
query data interactively in the same storage location where data is prepared. This means that customers do
not have to move their data from a Hadoop cluster to another analytic engine for data-warehousing scenarios.
Using the ORC file format, queries can use advanced joins, aggregations, and other advanced Hive
optimizations against the same data that was created in the data-preparation phase.
1.1 The importance of fast queries in big data When executing a query in database engines like Microsoft® SQL Server or Oracle®, it can be expensive to
run queries for the first time, until the cache has been set up. Once the cache is ready, query speed can
increase dramatically. When running a Hive distributed query using the Tez engine, it may spin up containers
in the YARN resource manager to process data in the cluster. This process is relatively expensive to start,
and even though there is an option for the Tez container to reuse it, it is not caching fragments of the results
or query access patterns for use across multiple sessions that SQL Server and other relational database
engines provide.
One of the key challenges has been the coarse-grained resource-sharing model that Hadoop uses. While this
allows extremely high throughput on long-running batch jobs, it struggles to scale down to interactive
performance across many active users. Situations in which there are many smaller queries requested often
from the system constitute the bulk of data science use cases. LLAP solves this challenge by using its own
resource management and pre-emption policies within the resources granted to it by YARN. Before LLAP, if a
high-priority job appears, entire YARN containers would need to be pre-empted, potentially losing minutes or
hours of work. Pre-emption in LLAP works at the query fragment level (less than a query) and means that
fast queries can run even while long-running queries run in the background.
LLAP offers the following advantages:
• Can serve more jobs or users from the same size of cluster
• Can speed up performance by the following factors:
- 7x to 10x on smaller queries, aggregations, and queries that select a small slice of data, even if
the logic is complex
- 2x to 3x for large queries
- 1.5x to 2x for queries dominated by large output
1.2 Why LLAP is used Hive has sped up enormously over the last couple of years with help from Tez, the cost-based optimizer, and
ORC files. LLAP aims to push latencies into the sub-second range. LLAP sub-second queries allow users to
deploy Hive for interactive dashboards and explorative analytics that have more demanding response-time
requirements. However, startup costs are a bottleneck (JVM takes 100s of milliseconds to start up) and
reading from HDFS or other deeper storages are relatively expensive.
Apache Hive 3 with LLAP
7 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
1.3 How LLAP makes queries easier LLAP enables sub-second SQL analytics on Hadoop by intelligently caching data in memory with persistent
servers that instantly process SQL queries. Sub-second queries require fast query execution and low setup
cost. LLAP on Hive brings many enhancements to the hive execution engine like Smarter Map Joins, Better
Map Join Vectorization, a fully vectorized pipeline, and a smarter CBO. The LLAP daemon has several
executors that execute work as fragments. These work queues have pluggable priority which prioritizes low
latency queries over long running queries. Also, lower–priority fragments can be pre-empted. For example, a
fragment can start running before inputs are ready, resulting in better pipelining, A fragment is able to
complete if all source data is ready. If the data is not ready, it may never free the executor. Fragments that
cannot complete may be pre-empted, which improves throughput and prevents deadlocks.
Currently, Hive I/O and input decoding is interleaved with processing. Remote HDFS reads are expensive
even if performed on local disks, and data compression and decoding are expensive with I/O elevator.
Reading, decoding, and processing are parallel depending on the workload, I/O, and processing.
Asynchronous I/O and efficient in-memory caching-reduces I/O cost and parallelizes I/O processing, pre-
fetching, and multi-threaded processing.
1.4 Apache Hive with LLAP architecture overview LLAP is a new hybrid execution model that enables efficiencies across queries, such as caching of columnar-
data, JIT-friendly operator pipelines. This hybrid model consists of a long-lived service interacting with on-
demand elastic containers serving as a tightly integrated DAG-based framework for query execution.
Figure 1 depicts an architecture overview of Apache Hive with LLAP.
Apache Hive with LLAP architecture overview
Apache Hive 3 with LLAP
8 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
This model enables the following:
• Combines daemons and containers for fast, concurrent execution of analytical workloads
• Replaces direct interactions with the HDFS data node
• Eliminates startup costs for tasks
• Performs concurrent queries without specialized YARN queue setup
• Enables sharing of metadata, map join tables
• Allows LLAP daemons to run on existing YARN
• Uses Apache slider for provisioning and recovery which is easy to bring up, tear down, and share
clusters
• Accesses LLAP daemons using a fragment-oriented API that is geared toward handling the chunks of
computation — input, output, processing operations, and metadata
Small/short queries are largely processed by this daemon directly, while larger queries will be performed in
standard YARN containers. LLAP processing modes are flexible: A job can run entirely in Tez, the processing
can be divided between Tez containers and LLAP executors, or it can run entirely within LLAP. LLAP can be
turned on and off, either for the whole cluster or per job. The Hive client decides where query fragments run,
either all in LLAP, none in LLAP, or in an intelligent mix. Tez continues to manage the coordination of the Hive
processing DAG with or without LLAP as shown in the Figure 2.
Comparision of MapReduce, Tez, and Tez with LLAP processes
Any request to an LLAP node contains the data location and metadata. It processes local and remote
locations; locality is the caller’s responsibility (YARN). Failure and recovery is simplified because any data
node can still be used to process any fragment of the input data. The Tez AM can thus simply rerun failed
fragments on the cluster.
1.5 Optimizations One of the most important and best-known LLAP optimizations is the ability to keep data live in memory so
that it can be used in multiple queries. In addition to reducing disk-I/O, this saves the considerable CPU cost
Apache Hive 3 with LLAP
9 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
of decompressing and un-marshalling the data repeatedly. Unlike MR or Tez processing, which typically
interleave I/O operations and processing, LLAP uses a multi-threaded I/O elevator model which executes
data reads, decoding, and processing on separate threads to keep the processing threads busier. Processing
multiple fragments within a single daemon means that the cost of JIT compiling is amortized over more
executions of the same code. In addition to sequential sharing (caching), stateless daemons can share
access to data among multiple parallel threads. Metadata and indexes are also cached and shared across
multiple fragments.
1.6 Benefits of LLAP LLAP along with Apache Ranger enable fine-grained security for the Hadoop ecosystem, including data
masking and filtering, by providing interfaces for external clients like Spark to read. LLAP uses persistent
query servers to avoid long startup times and deliver fast SQL. It also shares its in-memory cache among all
SQL users, maximizing the use of this scarce resource. LLAP has fine-grained resource management and
pre-emption, making it great for highly concurrent access across many users. Additionally, is 100%
compatible with existing Hive SQL and Hive tools.
Dell EMC Isilon F800 all-flash NAS
10 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
2 Dell EMC Isilon F800 all-flash NAS Dell EMC Isilon F800 all-flash scale-out NAS storage provides up to 250,000 IOPS and 15 GB/s bandwidth
per chassis. With a choice of SSD drive capacities, all-flash storage ranges from 96 TB to 924 TB per chassis
making the Isilon F800 ideal for demanding storage requirements in high-volume Hadoop data applications.
In additional to the all-flash, high-performance, scale-out hardware design of the Isilon F800, the embedded
storage operating system (Isilon OneFS) provides a unifying clustered file system with built-in scalable data
protection that simplifies storage management and administration. OneFS is a fully symmetric file system with
no single point of failure — taking advantage of clustering not just to scale performance and capacity, but also
to allow for any-to-any failover and multiple levels of redundancy that go far beyond the capabilities of RAID.
OneFS allows hardware to be incorporated or removed from the cluster at will and at any time, abstracting the
data and applications away from the hardware. Data is given infinite longevity and the cost and pain of data
migrations and hardware refreshes are eliminated.
2.1 Isilon nodes OneFS works exclusively with the Isilon scale-out NAS nodes, referred to as a cluster. A single Isilon cluster
consists of multiple nodes, which are rack-mountable enterprise appliances containing memory, CPU,
networking, Ethernet or low-latency InfiniBand interconnects, disk controllers, and storage media. As such,
each node in the distributed cluster has compute as well as storage capabilities.
With the new generation of Isilon hardware (Gen 6), a single chassis of 4 nodes in a 4U form factor is
required to create a cluster, which currently scales up to 144 nodes. Previous Isilon hardware platforms need
a minimum of three nodes and 6U of rack space to form a cluster. There are several types of nodes, all of
which can be incorporated into a single cluster, where different nodes provide varying ratios of capacity to
throughput or IOPS. This provides customers the ability to tier data and meet price and performance
requirements by using different Isilon storage node types in the storage cluster.
Each node or chassis added to a cluster increases the aggregate disk, cache, CPU, and network capacity.
OneFS leverages each of the hardware building blocks so that the whole becomes greater than the sum of
the parts. The RAM is grouped together into a single coherent cache, allowing I/O on any part of the cluster to
benefit from data cached anywhere. A file system journal ensures that writes are safe across power failures.
Spindles and CPUs are combined to increase throughput, capacity, and IOPS as the cluster grows, for
access to one file or for multiple files. A cluster’s storage capacity can range from a minimum of 18 TB to a
maximum of approximately 68 PB. The maximum capacity will continue to increase as disk drives and node
chassis continue to get denser.
Isilon nodes are broken into several classes, or tiers, according to their functionality:
Tier I/O profile Drive media Isilon node types
Extreme Performance High performance, low latency Flash F800
Performance Transactional I/O SAS and SSD H600, S210
Hybrid/Utility Concurrency and streaming throughput SATA/SAS and SSD
H500, H400, X410, HD400
Archive Nearline and deep archive SATA A200, A2000, NL410, HD400
Dell EMC Isilon F800 all-flash NAS
11 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
This paper focuses on the F800 node type for Hadoop. A good alternative to the F800 is the H600 node type
if storage capacity requirements are lower.
2.2 Network There are two types of networks associated with a cluster, internal and external, which are described in the
following subsections.
2.2.1 Back-end (internal) network All intra-node communication in a storage cluster is performed across a dedicated back-end network,
comprising either 10 GbE or 40 GbE Ethernet, or low-latency QDR InfiniBand (IB). This back-end network,
which is configured with redundant switches for high availability, acts as the backplane for the storage cluster.
This enables each storage node to act as a contributor in the storage cluster and isolates node-to-node
communication to a private, high-speed, low-latency network. This back-end network utilizes the Internet
Protocol (IP) for node-to-node communication.
2.2.2 Front-end (external) network Clients connect to the storage cluster using Ethernet connections (1 GbE, 10 GbE, or 40 GbE) that are
available on all storage nodes. Because each storage node provides its own Ethernet ports, the amount of
network bandwidth available to the storage cluster scales linearly with performance and capacity. The Isilon
storage cluster supports standard network communication protocols to a customer network, including NFS,
SMB, HTTP, FTP, and HDFS. Additionally, OneFS provides full integration with both IPv4 and IPv6
environments.
2.3 File system structure The OneFS file system is based on the UNIX file system (UFS) and is a very fast distributed file system. Each
cluster creates a single namespace and file system. This means that the file system is distributed across all
nodes in the cluster and is accessible by clients connecting to any node in the cluster. There is no partitioning
and no need for volume creation.
Because all information is shared among nodes across the internal network, data can be written to or read
from any node, thus optimizing performance when multiple Hadoop users or applications are concurrently
reading and writing to the same set of data.
For more details on Isilon and OneFS see the document Dell EMC Isilon OneFS: A Technical Overview.
2.4 Why use Dell EMC Isilon for big data and analytics The Isilon scale-out NAS platform provides Apache Hive and Hadoop clients with direct access to big data
through a Hadoop File System (HDFS) interface. Powered by the distributed Isilon OneFS operating system,
an Isilon cluster delivers a scalable pool of storage with a global namespace.
Hive clients access data that is stored in an Isilon cluster by using the HDFS protocols. Every Isilon node can
act as a NameNode and a DataNode for a Hadoop cluster. Each node boosts performance and expands the
cluster's capacity. For big data analytics, the Isilon scale-out distributed architecture minimizes bottlenecks,
rapidly serves big data, and optimizes performance for analytic jobs.
Dell EMC Isilon F800 all-flash NAS
12 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
An Isilon cluster fosters data analytics without ingesting data into an HDFS-based file system. An Isilon
cluster allows storing data on an enterprise storage platform with existing workflows and standard protocols,
including SMB, HTTP, FTP, REST, and NFS, as well as HDFS. Regardless of whether the data is written with
SMB or NFS, it can be analyzed with either Hadoop compute cluster through HDFS. There is no need to set
up an HDFS file system and then load data into it with tedious HDFS copy commands or inefficient Hadoop
connectors.
An Isilon cluster simplifies data management while cost-effectively maximizing the value of data. Although
high-performance computing with Hadoop has traditionally stored data locally in the compute cluster’s HDFS
file system, the following use cases make a compelling case for coupling Hadoop- or Hive-based analytics
with Isilon scale-out NAS:
• Store data in a POSIX-compliant file system with SMB and NFS workflows and then access it through
HDFS
• Scale storage independently of compute as data sets grow
• Protect data more reliably and efficiently instead of replicating it with HDFS 3X mirror replication
• Eliminate HDFS copy operations to ingest data and Hadoop fs commands to manage data
• Implement distributed fault-tolerant NameNode and DataNode services
• Manage data with enterprise storage features such as deduplication and snapshots
Storing data in an Isilon scale-out NAS cluster instead of HDFS clients streamlines the entire analytics
workflow. The Isilon OneFS HDFS interface eliminates extracting the data from a storage system and loading
it into an HDFS file system. Isilon OneFS multiprotocol data access with SMB and NFS eliminates exporting
the data after analyzing it. The result not only increases the ease and flexibility of analyzing data, but also can
reduce capital expenditures and operating expenses.
Solution overview
13 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
3 Solution overview
3.1 Isilon for big data analytics using Apache Hive The Isilon scale-out platform combines modular hardware with unified software to provide the storage
foundation for data analysis. Isilon scale-out NAS is a fully distributed system that consists of nodes of
modular hardware arranged in a cluster. The distributed Isilon OneFS operating system combines the
memory, I/O, CPUs, and disks of the nodes into a cohesive storage unit to present a global namespace as a
single file system.
The nodes work together as peers in a shared-nothing hardware architecture with no single point of failure.
Every node adds capacity, performance, and resiliency to the cluster, and each node acts as a NameNode
and DataNode. The NameNode daemon is a distributed process that runs on all the nodes in the cluster. A
compute client can connect to any node in the cluster to access NameNode services.
As nodes are added, the file system expands dynamically and redistributes data, eliminating the work of
partitioning disks and creating volumes. The result is a highly efficient and resilient storage architecture that
brings all the advantages of an enterprise scale-out NAS system to storing data for analysis.
Unlike traditional storage, the Hive ratio of CPU, RAM, and disk space depends on the workload — factors
that make it difficult to size a Hadoop cluster before having a chance to measure the Hive workload.
Expanding data sets also makes sizing decisions up front problematic. Isilon scale-out NAS lends itself
perfectly to this scenario: Isilon scale-out NAS allows increasing CPUs, RAM, and disk space by adding
nodes to dynamically match storage capacity and performance with the demands of a dynamic Hive
workload.
An Isilon cluster optimizes data protection. OneFS more efficiently and reliably protects data than HDFS. The
HDFS file system, by default, replicates a block of data three times. In contrast, OneFS stripes the data
across the cluster and protects the data with forward error correction codes, which consume less space than
replication with better protection.
An Isilon cluster also includes enterprise features to back up your data and to provide high availability. For
example, in managing DataNode data, a best practice with a traditional Hadoop system is to back up the data
to another system — an operation that must be performed with brute force by using a tool like DistCP. OneFS
includes support for NDMP backups, cluster synchronization, geo-replication, snapshots, file system journal,
virtual hot spare, antivirus, Integrity Scan, dynamic sector repair, and accelerated drive rebuilds.
The enterprise features of OneFS ease data management. OneFS includes storage pools, deduplication,
automated tiering, quotas, high-performing SSDs, capacity-optimized HDDs, and cluster monitoring and
forecasting with InsightIQ.
SmartPools, for example, provide tiered storage for storing current data in a high-performance storage pool
while storing older data in a lower, more cost-effective pool in case it needs to be analyzed again later.
For security, OneFS can authenticate HDFS connections with Kerberos and includes support for Ranger.
SmartLock can protect sensitive data from malicious, accidental, or premature alteration or deletion to help
comply with SEC 17a-4 regulations.
Solution overview
14 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
3.2 Hortonworks HDP 3.0.1 with Apache Hive 3 and Isilon topology The Apache Hive and Isilon F800 cluster tested in this paper uses HDFS as the network communication
protocol with a 40 GbE front-end and back-end network. The Hadoop compute cluster consists of twelve
servers with single 10 GbE interfaces that connect to the front-end network. The Hadoop compute cluster
runs the Hive applications and clients, and the Isilon cluster stores and serves all the data for the
environment.
A high-level example of a Hortonworks HDP 3.0.1 and Isilon topology with combined hardware, software, and
networks is shown in Figure 3.
HDP 3.0.1 and Isilon topology
Tested configuration
15 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
4 Tested configuration
4.1 Hadoop cluster The Hortonworks HDP 3.0.1 package containing Apache Hadoop 3.1.1 and Apache Hive 3.1.0 were tested in
this paper. Detailed configurations of YARN, TEZ, Hive, and YARN Queue are shown in appendix A.
4.2 Compute nodes All compute nodes are identical generic servers with 20 cores, 256 GB RAM, and a 10 Gb NIC, and are
running CentOS Linux release 7.4.1708 (Core).
A total of 12 compute servers were used to build a Hadoop cluster of three master and nine worker nodes.
The same Hadoop cluster was used for Hive TPC-DS test scenarios that are described in detail in section 5.
4.3 Isilon F800: presented as NameNode/DataNode to Hadoop cluster Two Isilon F800 chassis (eight nodes total) with a total of 120 x 1.6 TB SSDs were used for Hive LLAP +
Isilon testing. Each node has one 40 GbE connection to the front-end client network and one 40 GbE
connection to the private back-end network. The configuration details are as follows:
• Isilon model tested: Isilon F800-4U-Single-256GB-1x1GE-2x40GE SFP+-24TB SSD
• Isilon OneFS release tested: OneFS v 8.1.2.0 (HDFS configuration details are listed in appendix A)
4.4 Hive testbench For Hive, Hortonworks provides a project which was used to run the TPC-DS benchmark on Hive. All 99 TPC-
DS queries were present in the project. The full set of trivially-modified queries used in the remainder of this
paper are in listed the updated Hortonworks testbench repository. The TPC-DS benchmark data models are
modeled after the decision support functions of a retail product supplier. TPC-DS consists of seven fact tables
and 17 dimensions.
TPC-DS consists of 99 queries which are divided into four broad classes:
• Reporting Queries
• Ad-hoc Queries
• Iterative OLAP Queries
• Data Mining Queries
Two dataset-size ORC file format databases were tested: 20 GB and 3 TB. All 99 TPC-DS queries ran
sequentially against the 20 GB and 3 TB databases residing on Isilon storage.
TPC-DS ANSI SQL compliance and performance test results
16 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
5 TPC-DS ANSI SQL compliance and performance test results
5.1 HDP 3.0.1 configuration The following tests used the same HDP cluster to run Hive on TEZ and LLAP. Table 1 details the test
environment.
Test environment
Master node Worker node Storage Data size
3 9 Isilon F800 3 TB TPC-DS
20 cores 256 GB RAM 10 Gb NIC CentOS Linux release 7.4.1708
20 cores 256 GB RAM 10 Gb NIC CentOS Linux release 7.4.1708
Isilon F800-4U-Single-256GB-1x1GE-2x40GE SFP+-24TB SSD
Note: Tests were performed using the default out-of-the-box configurations resulting in no optimizations, no
special settings, and no query change for any engine.
5.2 Hive LLAP SQL compliance test HDP 3.0.1 with Hive 3 can run all 99 TPC-DS queries for the data stored on the Isilon OneFS system with
only trivial modifications (defined as simple, mechanical rewrites such as changing column names or aliases,
adding columns to the select list, and other simple transformations.) For reference, the full set of trivially-
modified queries used in the remainder of this paper are in the updated Hortonworks testbench repository.
Table 2 lists the successful execution of all 99 TPC-DS queries and the runtime, which shows the ANSI SQL
compliance of Hive LLAP on Isilon OneFS 8.1.2.1
Hive LLAP TPD-DC test: 99 queries showing execution time with scale factor of 20
Query # Time(s) Query # Time(s) Query # Time(s) Query # Time(s) Query # Time(s)
1.sql 16 21.sql 15 41.sql 7 61.sql 9 81.sql 13
2.sql 26 22.sql 11 42.sql 8 62.sql 16 82.sql 12
3.sql 8 23.sql 70 43.sql 8 63.sql 9 83.sql 14
4.sql 19 24.sql 46 44.sql 14 64.sql 35 84.sql 17
5.sql 27 25.sql 15 45.sql 9 65.sql 10 85.sql 37
6.sql 8 26.sql 10 46.sql 10 66.sql 14 86.sql 9
1 Based on Dell EMC internal testing, April 2019, running the TPC-DS benchmark on Isilon OneFS 8.1.2 with Hive. Actual results may vary.
TPC-DS ANSI SQL compliance and performance test results
17 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
Query # Time(s) Query # Time(s) Query # Time(s) Query # Time(s) Query # Time(s)
7.sql 10 27.sql 11 47.sql 14 67.sql 14 87.sql 18
8.sql 12 28.sql 17 48.sql 9 68.sql 9 88.sql 14
9.sql 42 29.sql 16 49.sql 43 69.sql 13 89.sql 8
10.sql 19 30.sql 15 50.sql 12 70.sql 12 90.sql 13
11.sql 21 31.sql 14 51.sql 16 71.sql 13 91.sql 9
12.sql 14 32.sql 17 52.sql 9 72.sql 52 92.sql 14
13.sql 12 33.sql 13 53.sql 9 73.sql 10 93.sql 24
14.sql 93 34.sql 11 54.sql 20 74.sql 13 94.sql 21
15.sql 9 35.sql 13 55.sql 11 75.sql 30 95.sql 27
16.sql 34 36.sql 10 56.sql 13 76.sql 18 96.sql 16
17.sql 18 37.sql 15 57.sql 12 77.sql 21 97.sql 18
18.sql 10 38.sql 13 58.sql 21 78.sql 28 98.sql 22
19.sql 12 39.sql 11 59.sql 12 79.sql 9 99.sql 26
20.sql 16 40.sql 25 60.sql 13 80.sql 34
5.3 Hive LLAP SQL performance test Figure 4 shows the 38 queries derived from the Hortonworks Hive testbench that ran Hive on TEZ and LLAP
engines successfully. These 38 queries are randomly selected from the four broad classes of reporting, ad-
hoc, iterative OLAP, and data mining queries of 99 TPC-DS queries.
Total runtime comparison of 38 queries
0
500
1000
1500
2000
2500
3000
3500
Total runtime (s) (lower is better)
Hive/TEZ Hive/LLAP Uncached Hive/LLAP Cached
Worker nodes: 9 File format: ORC Configuration: Default HDP 3.0.1
TPC-DS ANSI SQL compliance and performance test results
18 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
As shown, LLAP delivers a dramatic performance gain. The aggregated runtime of 38 queries on the
Hive/TEZ and Hive/LLAP engines has a substantial 2x performance boost in the case of uncached
Hive/LLAP, and a 12x decrease in times for cached Hive/LLAP.
Note: Hive on LLAP uncached is defined as when the compute resource memory is empty, and data is pulled
from the Isilon system for the first time. Hive LLAP cached is defined as when the compute resource memory
is populated with the dataset during interactive querying.
Figure 5 shows Hive LLAP performance compared to Hive TEZ for the database stored on the Isilon OneFS
system. In this case, Hive on LLAP ran uncached (with an empty compute resource cache), and the average
speed increase is 2x, proving that Isilon OneFS supports Hive LLAP with a performance boost compared to
Hive on the TEZ engine.
Hive/LLAP uncached shows a 2x runtime performance boost compared to Hive/TEZ
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0
50
100
150
200
250
query
1.s
ql
query
3.s
ql
query
6.s
ql
query
8.s
ql
query
12.s
ql
query
15.s
ql
query
20.s
ql
query
21.s
ql
query
22.s
ql
query
26.s
ql
query
30.s
ql
query
32.s
ql
query
33.s
ql
query
34.s
ql
query
37.s
ql
query
39.s
ql
query
40.s
ql
query
41.s
ql
query
42.s
ql
query
43.s
ql
query
45.s
ql
query
48.s
ql
query
52.s
ql
query
53.s
ql
query
55.s
ql
query
56.s
ql
query
60.s
ql
query
66.s
ql
query
68.s
ql
query
71.s
ql
query
73.s
ql
query
79.s
ql
query
83.s
ql
query
84.s
ql
query
85.s
ql
query
86.s
ql
query
90.s
ql
query
91.s
ql
query
98.s
ql
Spped incre
ase (
TE
Z/L
LA
P)
(big
ger
is b
etter)
Runtim
e (
s)
(log.
scale
) (low
er
is b
etter) TPC-DS queries,
3TB, 9 nodes
Hive/TEZ (s) Hive/LLAP Uncached Speedup
2x average speed increase
TPC-DS ANSI SQL compliance and performance test results
19 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
Figure 6 displays the performance for Hive LLAP cached (the compute resource cached data set from the
previous executions). This shows a 13x performance boost compared to Hive on the TEZ engine. Hive LLAP
data is cached during interactive querying on the same dataset, and the performance drastically increases in
Hive LLAP due to the hot cache on compute nodes. This minimizes the data transfer between storage and
compute, and enables sub-second queries in Hive by keeping all data and servers running and in-memory all
the time, while retaining to ability for elastic scalability within the YARN cluster.
Hive/LLAP cached shows a 13x runtime performance boost compared to Hive/TEZ
LLAP, along with Apache Ranger, enables fine-grained security for the Hadoop ecosystem, including data
masking and filtering, by providing interfaces for external clients like Spark to read. LLAP is ideal for Hadoop
clusters with decoupled compute and a storage-like cloud because it caches data in memory and keeps it
compressed, overcoming long cloud-storage access times and stretching the amount of data that can fit the
compute cluster memory.
0
5
10
15
20
25
30
35
0
50
100
150
200
250query
1.s
ql
query
3.s
ql
query
6.s
ql
query
8.s
ql
query
12.s
ql
query
15.s
ql
query
20.s
ql
query
21.s
ql
query
22.s
ql
query
26.s
ql
query
30.s
ql
query
32.s
ql
query
33.s
ql
query
34.s
ql
query
37.s
ql
query
39.s
ql
query
40.s
ql
query
41.s
ql
query
42.s
ql
query
43.s
ql
query
45.s
ql
query
48.s
ql
query
52.s
ql
query
53.s
ql
query
55.s
ql
query
56.s
ql
query
60.s
ql
query
66.s
ql
query
68.s
ql
query
71.s
ql
query
73.s
ql
query
79.s
ql
query
83.s
ql
query
84.s
ql
query
85.s
ql
query
86.s
ql
query
90.s
ql
query
91.s
ql
query
98.s
ql
Speed incre
ase (
TE
Z/L
LA
P)
(big
ger
is b
etter)
Runtim
e (
s)
(log.
scale
) (low
er
is b
etter) TPC-DS queries,
3 TB, 9 nodes
Hive/TEZ (s) Hive/LLAP Cached SpeedUp
12.65x average speed increase
Conclusion
20 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
6 Conclusion The Hortonworks Hive LLAP has been designed to enable sub-second SQL analytics on Hadoop in
anticipation of vast increases in data volumes. The ability of OneFS to scale to multiple petabytes in a single
file system while delivering high performance I/O makes Isilon storage ideal for Hive, and offers optimal
workload portability if considering offloading workloads from a legacy EDW.
This paper shows that the Isilon F800 all-flash NAS storage solution meets Hive LLAP SQL compliance, and
performs very well under I/O load with a Hive Data warehouse for both small and large data sets. The
Hortonworks Hive Testbench results for all 99 TPC-DS queries are included in this paper and show excellent
results in terms of execution time for a 3 TB scale factor.
With Dell EMC Isilon storage, enterprise organizations and Hadoop/Hive administrators can effortlessly scale
from tens of terabytes to tens of petabytes within a single file system, single volume, and with a single point of
administration. Isilon storage delivers high performance and high throughput without adding management
complexity.
Configuration details
21 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
A Configuration details
A.1 Hadoop cluster configurations
Figure 7 shows the HDP 3.0.1 Hadoop cluster configuration and software version used in testing, and Figure
8 shows the Apache Ambari™ server version.
Hadoop cluster software versions
Apache Ambari server version
Configuration details
22 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
A.1.1 OneFS service configurations
Isilon OneFS is presented as a service to the HDP 3.0.1 Hadoop cluster. Figure 9 and Figure 10 show the
configuration set used in testing.
OneFS service advanced core-site.xml file configurations
OneFS service hdfs-site.xml and custom core-site.xml file configurations
Configuration details
23 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
A.1.2 YARN service configurations
Figure 11 and Figure 12 show the HDP 3.0.1 cluster YARN service configurations set used in testing.
Yarn service general configurations
Configuration details
24 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
Yarn configuration summary
Configuration details
25 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
A.1.3 Hive service configurations
Figure 13 through Figure 17 show the HDP 3.0.1 Hadoop cluster Hive service configuration set used in
testing.
TEZ general configurations
Configuration details
26 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
TEZ -site.xml configurations, all set to default
Hive service summary
Configuration details
27 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
Hive configuration setting
Configuration details
28 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
Hive configuration setting 2
Configuration details
29 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
A.1.4 YARN queue manager
Figure 18 shows the HDP 3.0.1 Hadoop cluster YARN queue manager configured used in testing.
LLAP queue setting
A.2 Isilon configuration
The following features were configured on the Isilon cluster. The Smart features shown are product
differentiators that significantly enhance data storage performance and resiliency.
• Enable SmartPools settings across all Isilon nodes.
• Enable SmartConnect to provide automatic client connection load-balancing and failover capabilities.
• Enable SmartCache for write performance and Streaming Access for Data Access Optimization.
• Use optimization for a streaming data-access pattern.
• Use 40 Gb/s external network ports for NFS connections and internal 40 Gb/s ports for the data
interconnect network.
• Increase network MTU to 9000 (jumbo frames) for both internal and external networks.
Configuration details
30 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
The storage pool is created on the SmartPools tab, allowing you to specify the Isilon nodes and protection
settings. Figure 19 shows the storage pool configured for the Hadoop and Isilon cluster.
Storage Pools setting
Figure 20 shows the internal and external network configuration set in the network configuration tab, where
you can specify MTU size, IP info, and DNS server information.
Isilon network settings
Configuration details
31 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
The SmartConnect information is configured within the pool properties. In the Hadoop Isilon cluster, the pool
name is pool0 and the SmartConnect information (with IP and DNS info blacked out) is shown in Figure 21
Isilon pool details
Note: Isilon storage provides two 40 GbE front-end and two 40GbE back-end ports with each node. The
Hadoop and Isilon cluster was only configured with one 40 GbE front-end port and one 40GbE back-end port
on each node during performance testing. With production deployments, use both front-end and back-end 40
GbE ports.
Configuration details
32 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
A.2.1 OneFS HDFS settings
Figure 22 shows the HDFS is configured to Isilon system zone, namely /ifs/hdfs-hdp301. The root hdfs path
was tested in this paper.
OneFS HDFS settings
Configuration details
33 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
A.2.2 OneFS TCP tuning
The default TCP stack of OneFS requires tuning for Hadoop and 40 GbE connectivity. The tuning needs to be
done within the CLI directly on Isilon. A tcptune.sh script is available at GitHub.
Run sh ./tcptune.sh Max to make the changes. An example script run is shown as follows:
Before changes:
isilon# sh ./tcptune.sh Max
Tuning TCP stack to Max
TCP sysctls before...
kern.ipc.maxsockbuf=2097152
net.inet.tcp.sendbuf_max=2097152
net.inet.tcp.recvbuf_max=2097152
net.inet.tcp.sendbuf_inc=8192
net.inet.tcp.recvbuf_inc=16384
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
efs.bam.coalescer.insert_hwm=209715200
efs.bam.coalescer.insert_lwm=178257920
After changes:
Apply tuning...
Value set successfully
Value set successfully
Value set successfully
Value set successfully
Value set successfully
Value set successfully
Value set successfully
Value set successfully
TCP sysctls after...
kern.ipc.maxsockbuf=104857600
net.inet.tcp.sendbuf_max=52428800
net.inet.tcp.recvbuf_max=52428800
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=32768
net.inet.tcp.sendspace=26214400
net.inet.tcp.recvspace=26214400
efs.bam.coalescer.insert_hwm=209715200
efs.bam.coalescer.insert_lwm=178257920
net.inet.tcp.mssdflt=8948
Technical support and resources
34 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672
B Technical support and resources
Dell.com/support is focused on meeting customer needs with proven services and support.
Storage technical documents and videos provide expertise that helps to ensure customer success on Dell
EMC storage platforms.
B.1 Related resources
See the following referenced or recommended resources:
• Hortonworks Hive LLAP
• Hortonworks Hive TPC-DS
• Hortonworks HDP 3.0.1
• Isilon technical Overview
• Hortonworks HDP 3.0.1 Install on Isilon OneFS8.1.2
top related