dell emc isilon onefs: apache hive low latency …...apache hive 3 with llap 7 dell emc isilon...

H17672

Technical White Paper

Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance Running sub-second SQL on Apache Hadoop with Dell EMC Isilon F800 all-flash scale-out NAS storage and Isilon OneFS 8.1.2

Abstract This paper describes the support and performance test results for running

Apache® Hive™ 3 low-latency analytical processing (LLAP) with Dell EMC™

Isilon™ F800 all-flash NAS storage for sub-second queries.

April 2019

Revisions

2 Dell EMC Isilon OneFS: Apache Hive Low Latency Analytical Processing Performance | H17672

Revisions

Date Description

April 2019 Initial release

Acknowledgements

Author: Kirankumar Bhusanurmath ([email protected]), Analytics Solution Architect

The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this

publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable software license.

Copyright © 2019 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its

subsidiaries. Other trademarks may be trademarks of their respective owners. [4/9/2019] [Technical White Paper] [H17672]

mailto:[email protected]

Table of contents


Table of contents

Revisions............................................................................................................................................................................. 2

Acknowledgements ............................................................................................................................................................. 2

Table of contents ................................................................................................................................................................ 3

Executive summary ............................................................................................................................................................. 5

1 Apache Hive 3 with LLAP ............................................................................................................................................. 6

1.1 The importance of fast queries in big data ......................................................................................................... 6

1.2 Why LLAP is used .............................................................................................................................................. 6

1.3 How LLAP makes queries easier ....................................................................................................................... 7

1.4 Apache Hive with LLAP architecture overview ................................................................................................... 7

1.5 Optimizations ...................................................................................................................................................... 8

1.6 Benefits of LLAP ................................................................................................................................................. 9

2 Dell EMC Isilon F800 all-flash NAS ............................................................................................................................ 10

2.1 Isilon nodes ....................................................................................................................................................... 10

2.2 Network ............................................................................................................................................................. 11

2.2.1 Back-end (internal) network .............................................................................................................................. 11

2.2.2 Front-end (external) network ............................................................................................................................ 11

2.3 File system structure ........................................................................................................................................ 11

2.4 Why use Dell EMC Isilon for big data and analytics ......................................................................................... 11

3 Solution overview ....................................................................................................................................................... 13

3.1 Isilon for big data analytics using Apache Hive ................................................................................................ 13

3.2 Hortonworks HDP 3.0.1 with Apache Hive 3 and Isilon topology ..................................................................... 14

4 Tested configuration ................................................................................................................................................... 15

4.1 Hadoop cluster.................................................................................................................................................. 15

4.2 Compute nodes ................................................................................................................................................ 15

4.3 Isilon F800: presented as NameNode/DataNode to Hadoop cluster ............................................................... 15

4.4 Hive testbench .................................................................................................................................................. 15

5 TPC-DS ANSI SQL compliance and performance test results .................................................................................. 16

5.1 HDP 3.0.1 configuration ................................................................................................................................... 16

5.2 Hive LLAP SQL compliance test ...................................................................................................................... 16

5.3 Hive LLAP SQL performance test .................................................................................................................... 17

6 Conclusion .................................................................................................................................................................. 20

A Configuration details ................................................................................................................................................... 21

A.1 Hadoop cluster configurations .......................................................................................................................... 21

A.1.1 OneFS service configurations .......................................................................................................................... 22

Table of contents


A.1.2 YARN service configurations ............................................................................................................................ 23

A.1.3 Hive service configurations ............................................................................................................................... 25

A.1.4 YARN queue manager ..................................................................................................................................... 29

A.2 Isilon configuration ............................................................................................................................................ 29

A.2.1 OneFS HDFS settings ...................................................................................................................................... 32

A.2.2 OneFS TCP tuning ........................................................................................................................................... 33

B Technical support and resources ............................................................................................................................... 34

B.1 Related resources............................................................................................................................................. 34

Executive summary


Executive summary

The most significant feature introduced in Apache® Hive 2 was LLAP (low-latency analytical processing).

LLAP enables sub-second SQL analytics on Apache Hadoop® by intelligently caching data in memory with

persistent servers that instantly process SQL queries. Since LLAP is an evolution of the Hive architecture, it

does all of this with the same comprehensive ANSI-standard SQL support and proven scale that Hive is

known for, enabling Hive to run all 99 TPC-DS queries with only trivial modifications to the original source

queries.

One of the most exciting new features of Dell EMC™ Isilon™ OneFS™ 8.1.2 from is the official support for

Hortonworks HDP 3.0.1 with Apache Hive 3 LLAP. This paper describes the official support and performance

test results (Apache Hive LLAP SQL compliant) for a Hortonworks HDP 3.0.1 cluster on generic servers for

compute and Isilon F800 for storage.

This paper also shows how the Isilon F800 scale-out NAS solution provides compliance to ANSI-standard

SQL for LLAP, excellent performance, and better storage utilization with less drives and a smaller storage foot

print than legacy solutions.

Apache Hive 3 with LLAP


1 Apache Hive 3 with LLAP One common use case in enterprise computing is the repeated use of the same tables. Business queries tend

to focus on operations like aggregation and on extracting a small amount of data out of a large data set. In

recent years, most Hadoop customers use a rudimentary approach of moving data from a Hadoop cluster to a

relational database for interactive querying. This introduces a large latency as well as additional cost for

management, and makes maintenance of multiple analytic solutions difficult. LLAP allows data scientists to

query data interactively in the same storage location where data is prepared. This means that customers do

not have to move their data from a Hadoop cluster to another analytic engine for data-warehousing scenarios.

Using the ORC file format, queries can use advanced joins, aggregations, and other advanced Hive

optimizations against the same data that was created in the data-preparation phase.

1.1 The importance of fast queries in big data When executing a query in database engines like Microsoft® SQL Server or Oracle®, it can be expensive to

run queries for the first time, until the cache has been set up. Once the cache is ready, query speed can

increase dramatically. When running a Hive distributed query using the Tez engine, it may spin up containers

in the YARN resource manager to process data in the cluster. This process is relatively expensive to start,

and even though there is an option for the Tez container to reuse it, it is not caching fragments of the results

or query access patterns for use across multiple sessions that SQL Server and other relational database

engines provide.

One of the key challenges has been the coarse-grained resource-sharing model that Hadoop uses. While this

allows extremely high throughput on long-running batch jobs, it struggles to scale down to interactive

performance across many active users. Situations in which there are many smaller queries requested often

from the system constitute the bulk of data science use cases. LLAP solves this challenge by using its own

resource management and pre-emption policies within the resources granted to it by YARN. Before LLAP, if a

high-priority job appears, entire YARN containers would need to be pre-empted, potentially losing minutes or

hours of work. Pre-emption in LLAP works at the query fragment level (less than a query) and means that

fast queries can run even while long-running queries run in the background.

LLAP offers the following advantages:

• Can serve more jobs or users from the same size of cluster

• Can speed up performance by the following factors:

- 7x to 10x on smaller queries, aggregations, and queries that select a small slice of data, even if

the logic is complex

- 2x to 3x for large queries

- 1.5x to 2x for queries dominated by large output

1.2 Why LLAP is used Hive has sped up enormously over the last couple of years with help from Tez, the cost-based optimizer, and

ORC files. LLAP aims to push latencies into the sub-second range. LLAP sub-second queries allow users to

deploy Hive for interactive dashboards and explorative analytics that have more demanding response-time

requirements. However, startup costs are a bottleneck (JVM takes 100s of milliseconds to start up) and

reading from HDFS or other deeper storages are relatively expensive.



1.3 How LLAP makes queries easier LLAP enables sub-second SQL analytics on Hadoop by intelligently caching data in memory with persistent

servers that instantly process SQL queries. Sub-second queries require fast query execution and low setup

cost. LLAP on Hive brings many enhancements to the hive execution engine like Smarter Map Joins, Better

Map Join Vectorization, a fully vectorized pipeline, and a smarter CBO. The LLAP daemon has several

executors that execute work as fragments. These work queues have pluggable priority which prioritizes low

latency queries over long running queries. Also, lower–priority fragments can be pre-empted. For example, a

fragment can start running before inputs are ready, resulting in better pipelining, A fragment is able to

complete if all source data is ready. If the data is not ready, it may never free the executor. Fragments that

cannot complete may be pre-empted, which improves throughput and prevents deadlocks.

Currently, Hive I/O and input decoding is interleaved with processing. Remote HDFS reads are expensive

even if performed on local disks, and data compression and decoding are expensive with I/O elevator.

Reading, decoding, and processing are parallel depending on the workload, I/O, and processing.

Asynchronous I/O and efficient in-memory caching-reduces I/O cost and parallelizes I/O processing, pre-

fetching, and multi-threaded processing.

1.4 Apache Hive with LLAP architecture overview LLAP is a new hybrid execution model that enables efficiencies across queries, such as caching of columnar-

data, JIT-friendly operator pipelines. This hybrid model consists of a long-lived service interacting with on-

demand elastic containers serving as a tightly integrated DAG-based framework for query execution.

Figure 1 depicts an architecture overview of Apache Hive with LLAP.

Apache Hive with LLAP architecture overview



This model enables the following:

• Combines daemons and containers for fast, concurrent execution of analytical workloads

• Replaces direct interactions with the HDFS data node

• Eliminates startup costs for tasks

• Performs concurrent queries without specialized YARN queue setup

• Enables sharing of metadata, map join tables

• Allows LLAP daemons to run on existing YARN

• Uses Apache slider for provisioning and recovery which is easy to bring up, tear down, and share

clusters

• Accesses LLAP daemons using a fragment-oriented API that is geared toward handling the chunks of

computation — input, output, processing operations, and metadata

Small/short queries are largely processed by this daemon directly, while larger queries will be performed in

standard YARN containers. LLAP processing modes are flexible: A job can run entirely in Tez, the processing

can be divided between Tez containers and LLAP executors, or it can run entirely within LLAP. LLAP can be

turned on and off, either for the whole cluster or per job. The Hive client decides where query fragments run,

either all in LLAP, none in LLAP, or in an intelligent mix. Tez continues to manage the coordination of the Hive

processing DAG with or without LLAP as shown in the Figure 2.

Comparision of MapReduce, Tez, and Tez with LLAP processes

Any request to an LLAP node contains the data location and metadata. It processes local and remote

locations; locality is the caller’s responsibility (YARN). Failure and recovery is simplified because any data

node can still be used to process any fragment of the input data. The Tez AM can thus simply rerun failed

fragments on the cluster.

1.5 Optimizations One of the most important and best-known LLAP optimizations is the ability to keep data live in memory so

that it can be used in multiple queries. In addition to reducing disk-I/O, this saves the considerable CPU cost



of decompressing and un-marshalling the data repeatedly. Unlike MR or Tez processing, which typically

interleave I/O operations and processing, LLAP uses a multi-threaded I/O elevator model which executes

data reads, decoding, and processing on separate threads to keep the processing threads busier. Processing

multiple fragments within a single daemon means that the cost of JIT compiling is amortized over more

executions of the same code. In addition to sequential sharing (caching), stateless daemons can share

access to data among multiple parallel threads. Metadata and indexes are also cached and shared across

multiple fragments.

1.6 Benefits of LLAP LLAP along with Apache Ranger enable fine-grained security for the Hadoop ecosystem, including data

masking and filtering, by providing interfaces for external clients like Spark to read. LLAP uses persistent

query servers to avoid long startup times and deliver fast SQL. It also shares its in-memory cache among all

SQL users, maximizing the use of this scarce resource. LLAP has fine-grained resource management and

pre-emption, making it great for highly concurrent access across many users. Additionally, is 100%

compatible with existing Hive SQL and Hive tools.

Dell EMC Isilon F800 all-flash NAS


2 Dell EMC Isilon F800 all-flash NAS Dell EMC Isilon F800 all-flash scale-out NAS storage provides up to 250,000 IOPS and 15 GB/s bandwidth

per chassis. With a choice of SSD drive capacities, all-flash storage ranges from 96 TB to 924 TB per chassis

making the Isilon F800 ideal for demanding storage requirements in high-volume Hadoop data applications.

In additional to the all-flash, high-performance, scale-out hardware design of the Isilon F800, the embedded

storage operating system (Isilon OneFS) provides a unifying clustered file system with built-in scalable data

protection that simplifies storage management and administration. OneFS is a fully symmetric file system with

no single point of failure — taking advantage of clustering not just to scale performance and capacity, but also

to allow for any-to-any failover and multiple levels of redundancy that go far beyond the capabilities of RAID.

OneFS allows hardware to be incorporated or removed from the cluster at will and at any time, abstracting the

data and applications away from the hardware. Data is given infinite longevity and the cost and pain of data

migrations and hardware refreshes are eliminated.

2.1 Isilon nodes OneFS works exclusively with the Isilon scale-out NAS nodes, referred to as a cluster. A single Isilon cluster

consists of multiple nodes, which are rack-mountable enterprise appliances containing memory, CPU,

networking, Ethernet or low-latency InfiniBand interconnects, disk controllers, and storage media. As such,

each node in the distributed cluster has compute as well as storage capabilities.

With the new generation of Isilon hardware (Gen 6), a single chassis of 4 nodes in a 4U form factor is

required to create a cluster, which currently scales up to 144 nodes. Previous Isilon hardware platforms need

a minimum of three nodes and 6U of rack space to form a cluster. There are several types of nodes, all of

which can be incorporated into a single cluster, where different nodes provide varying ratios of capacity to

throughput or IOPS. This provides customers the ability to tier data and meet price and performance

requirements by using different Isilon storage node types in the storage cluster.

Each node or chassis added to a cluster increases the aggregate disk, cache, CPU, and network capacity.

OneFS leverages each of the hardware building blocks so that the whole becomes greater than the sum of

the parts. The RAM is grouped together into a single coherent cache, allowing I/O on any part of the cluster to

benefit from data cached anywhere. A file system journal ensures that writes are safe across power failures.

Spindles and CPUs are combined to increase throughput, capacity, and IOPS as the cluster grows, for

access to one file or for multiple files. A cluster’s storage capacity can range from a minimum of 18 TB to a

maximum of approximately 68 PB. The maximum capacity will continue to increase as disk drives and node

chassis continue to get denser.

Isilon nodes are broken into several classes, or tiers, according to their functionality:

Tier I/O profile Drive media Isilon node types

Extreme Performance High performance, low latency Flash F800

Performance Transactional I/O SAS and SSD H600, S210

Hybrid/Utility Concurrency and streaming throughput SATA/SAS and SSD

H500, H400, X410, HD400

Archive Nearline and deep archive SATA A200, A2000, NL410, HD400



This paper focuses on the F800 node type for Hadoop. A good alternative to the F800 is the H600 node type

if storage capacity requirements are lower.

2.2 Network There are two types of networks associated with a cluster, internal and external, which are described in the

following subsections.

2.2.1 Back-end (internal) network All intra-node communication in a storage cluster is performed across a dedicated back-end network,

comprising either 10 GbE or 40 GbE Ethernet, or low-latency QDR InfiniBand (IB). This back-end network,

which is configured with redundant switches for high availability, acts as the backplane for the storage cluster.

This enables each storage node to act as a contributor in the storage cluster and isolates node-to-node

communication to a private, high-speed, low-latency network. This back-end network utilizes the Internet

Protocol (IP) for node-to-node communication.

2.2.2 Front-end (external) network Clients connect to the storage cluster using Ethernet connections (1 GbE, 10 GbE, or 40 GbE) that are

available on all storage nodes. Because each storage node provides its own Ethernet ports, the amount of

network bandwidth available to the storage cluster scales linearly with performance and capacity. The Isilon

storage cluster supports standard network communication protocols to a customer network, including NFS,

SMB, HTTP, FTP, and HDFS. Additionally, OneFS provides full integration with both IPv4 and IPv6

environments.

2.3 File system structure The OneFS file system is based on the UNIX file system (UFS) and is a very fast distributed file system. Each

cluster creates a single namespace and file system. This means that the file system is distributed across all

nodes in the cluster and is accessible by clients connecting to any node in the cluster. There is no partitioning

and no need for volume creation.

Because all information is shared among nodes across the internal network, data can be written to or read

from any node, thus optimizing performance when multiple Hadoop users or applications are concurrently

reading and writing to the same set of data.

For more details on Isilon and OneFS see the document Dell EMC Isilon OneFS: A Technical Overview.

2.4 Why use Dell EMC Isilon for big data and analytics The Isilon scale-out NAS platform provides Apache Hive and Hadoop clients with direct access to big data

through a Hadoop File System (HDFS) interface. Powered by the distributed Isilon OneFS operating system,

an Isilon cluster delivers a scalable pool of storage with a global namespace.

Hive clients access data that is stored in an Isilon cluster by using the HDFS protocols. Every Isilon node can

act as a NameNode and a DataNode for a Hadoop cluster. Each node boosts performance and expands the

cluster's capacity. For big data analytics, the Isilon scale-out distributed architecture minimizes bottlenecks,

rapidly serves big data, and optimizes performance for analytic jobs.

http://www.emc.com/collateral/hardware/white-papers/h10719-isilon-onefs-technical-overview-wp.pdf



An Isilon cluster fosters data analytics without ingesting data into an HDFS-based file system. An Isilon

cluster allows storing data on an enterprise storage platform with existing workflows and standard protocols,

including SMB, HTTP, FTP, REST, and NFS, as well as HDFS. Regardless of whether the data is written with

SMB or NFS, it can be analyzed with either Hadoop compute cluster through HDFS. There is no need to set

up an HDFS file system and then load data into it with tedious HDFS copy commands or inefficient Hadoop

connectors.

An Isilon cluster simplifies data management while cost-effectively maximizing the value of data. Although

high-performance computing with Hadoop has traditionally stored data locally in the compute cluster’s HDFS

file system, the following use cases make a compelling case for coupling Hadoop- or Hive-based analytics

with Isilon scale-out NAS:

• Store data in a POSIX-compliant file system with SMB and NFS workflows and then access it through

HDFS

• Scale storage independently of compute as data sets grow

• Protect data more reliably and efficiently instead of replicating it with HDFS 3X mirror replication

• Eliminate HDFS copy operations to ingest data and Hadoop fs commands to manage data

• Implement distributed fault-tolerant NameNode and DataNode services

• Manage data with enterprise storage features such as deduplication and snapshots

Storing data in an Isilon scale-out NAS cluster instead of HDFS clients streamlines the entire analytics

workflow. The Isilon OneFS HDFS interface eliminates extracting the data from a storage system and loading

it into an HDFS file system. Isilon OneFS multiprotocol data access with SMB and NFS eliminates exporting

the data after analyzing it. The result not only increases the ease and flexibility of analyzing data, but also can

reduce capital expenditures and operating expenses.

Solution overview


3 Solution overview

3.1 Isilon for big data analytics using Apache Hive The Isilon scale-out platform combines modular hardware with unified software to provide the storage

foundation for data analysis. Isilon scale-out NAS is a fully distributed system that consists of nodes of

modular hardware arranged in a cluster. The distributed Isilon OneFS operating system combines the

memory, I/O, CPUs, and disks of the nodes into a cohesive storage unit to present a global namespace as a

single file system.

The nodes work together as peers in a shared-nothing hardware architecture with no single point of failure.

Every node adds capacity, performance, and resiliency to the cluster, and each node acts as a NameNode

and DataNode. The NameNode daemon is a distributed process that runs on all the nodes in the cluster. A

compute client can connect to any node in the cluster to access NameNode services.

As nodes are added, the file system expands dynamically and redistributes data, eliminating the work of

partitioning disks and creating volumes. The result is a highly efficient and resilient storage architecture that

brings all the advantages of an enterprise scale-out NAS system to storing data for analysis.

Unlike traditional storage, the Hive ratio of CPU, RAM, and disk space depends on the workload — factors

that make it difficult to size a Hadoop cluster before having a chance to measure the Hive workload.

Expanding data sets also makes sizing decisions up front problematic. Isilon scale-out NAS lends itself

perfectly to this scenario: Isilon scale-out NAS allows increasing CPUs, RAM, and disk space by adding

nodes to dynamically match storage capacity and performance with the demands of a dynamic Hive

workload.

An Isilon cluster optimizes data protection. OneFS more efficiently and reliably protects data than HDFS. The

HDFS file system, by default, replicates a block of data three times. In contrast, OneFS stripes the data

across the cluster and protects the data with forward error correction codes, which consume less space than

replication with better protection.

An Isilon cluster also includes enterprise features to back up your data and to provide high availability. For

example, in managing DataNode data, a best practice with a traditional Hadoop system is to back up the data

to another system — an operation that must be performed with brute force by using a tool like DistCP. OneFS

includes support for NDMP backups, cluster synchronization, geo-replication, snapshots, file system journal,

virtual hot spare, antivirus, Integrity Scan, dynamic sector repair, and accelerated drive rebuilds.

The enterprise features of OneFS ease data management. OneFS includes storage pools, deduplication,

automated tiering, quotas, high-performing SSDs, capacity-optimized HDDs, and cluster monitoring and

forecasting with InsightIQ.

SmartPools, for example, provide tiered storage for storing current data in a high-performance storage pool

while storing older data in a lower, more cost-effective pool in case it needs to be analyzed again later.

For security, OneFS can authenticate HDFS connections with Kerberos and includes support for Ranger.

SmartLock can protect sensitive data from malicious, accidental, or premature alteration or deletion to help

comply with SEC 17a-4 regulations.

Solution overview


3.2 Hortonworks HDP 3.0.1 with Apache Hive 3 and Isilon topology The Apache Hive and Isilon F800 cluster tested in this paper uses HDFS as the network communication

protocol with a 40 GbE front-end and back-end network. The Hadoop compute cluster consists of twelve

servers with single 10 GbE interfaces that connect to the front-end network. The Hadoop compute cluster

runs the Hive applications and clients, and the Isilon cluster stores and serves all the data for the

environment.

A high-level example of a Hortonworks HDP 3.0.1 and Isilon topology with combined hardware, software, and

networks is shown in Figure 3.

HDP 3.0.1 and Isilon topology

Tested configuration


4 Tested configuration

4.1 Hadoop cluster The Hortonworks HDP 3.0.1 package containing Apache Hadoop 3.1.1 and Apache Hive 3.1.0 were tested in

this paper. Detailed configurations of YARN, TEZ, Hive, and YARN Queue are shown in appendix A.

4.2 Compute nodes All compute nodes are identical generic servers with 20 cores, 256 GB RAM, and a 10 Gb NIC, and are

running CentOS Linux release 7.4.1708 (Core).

A total of 12 compute servers were used to build a Hadoop cluster of three master and nine worker nodes.

The same Hadoop cluster was used for Hive TPC-DS test scenarios that are described in detail in section 5.

4.3 Isilon F800: presented as NameNode/DataNode to Hadoop cluster Two Isilon F800 chassis (eight nodes total) with a total of 120 x 1.6 TB SSDs were used for Hive LLAP +

Isilon testing. Each node has one 40 GbE connection to the front-end client network and one 40 GbE

connection to the private back-end network. The configuration details are as follows:

• Isilon model tested: Isilon F800-4U-Single-256GB-1x1GE-2x40GE SFP+-24TB SSD

• Isilon OneFS release tested: OneFS v 8.1.2.0 (HDFS configuration details are listed in appendix A)

4.4 Hive testbench For Hive, Hortonworks provides a project which was used to run the TPC-DS benchmark on Hive. All 99 TPC-

DS queries were present in the project. The full set of trivially-modified queries used in the remainder of this

paper are in listed the updated Hortonworks testbench repository. The TPC-DS benchmark data models are

modeled after the decision support functions of a retail product supplier. TPC-DS consists of seven fact tables

and 17 dimensions.

TPC-DS consists of 99 queries which are divided into four broad classes:

• Reporting Queries

• Ad-hoc Queries

• Iterative OLAP Queries

• Data Mining Queries

Two dataset-size ORC file format databases were tested: 20 GB and 3 TB. All 99 TPC-DS queries ran

sequentially against the 20 GB and 3 TB databases residing on Isilon storage.

https://github.com/hortonworks/hive-testbench/tree/hdp26/sample-queries-tpcds

TPC-DS ANSI SQL compliance and performance test results


5 TPC-DS ANSI SQL compliance and performance test results

5.1 HDP 3.0.1 configuration The following tests used the same HDP cluster to run Hive on TEZ and LLAP. Table 1 details the test

environment.

Test environment

Master node Worker node Storage Data size

3 9 Isilon F800 3 TB TPC-DS

20 cores 256 GB RAM 10 Gb NIC CentOS Linux release 7.4.1708

20 cores 256 GB RAM 10 Gb NIC CentOS Linux release 7.4.1708

Isilon F800-4U-Single-256GB-1x1GE-2x40GE SFP+-24TB SSD

Note: Tests were performed using the default out-of-the-box configurations resulting in no optimizations, no

special settings, and no query change for any engine.

5.2 Hive LLAP SQL compliance test HDP 3.0.1 with Hive 3 can run all 99 TPC-DS queries for the data stored on the Isilon OneFS system with

only trivial modifications (defined as simple, mechanical rewrites such as changing column names or aliases,

adding columns to the select list, and other simple transformations.) For reference, the full set of trivially-

modified queries used in the remainder of this paper are in the updated Hortonworks testbench repository.

Table 2 lists the successful execution of all 99 TPC-DS queries and the runtime, which shows the ANSI SQL

compliance of Hive LLAP on Isilon OneFS 8.1.2.1

Hive LLAP TPD-DC test: 99 queries showing execution time with scale factor of 20

Query # Time(s) Query # Time(s) Query # Time(s) Query # Time(s) Query # Time(s)

1.sql 16 21.sql 15 41.sql 7 61.sql 9 81.sql 13






1 Based on Dell EMC internal testing, April 2019, running the TPC-DS benchmark on Isilon OneFS 8.1.2 with Hive. Actual results may vary.

https://github.com/hortonworks/hive-testbench/tree/hdp3/sample-queries-tpcds



Query # Time(s) Query # Time(s) Query # Time(s) Query # Time(s) Query # Time(s)














20.sql 16 40.sql 25 60.sql 13 80.sql 34

5.3 Hive LLAP SQL performance test Figure 4 shows the 38 queries derived from the Hortonworks Hive testbench that ran Hive on TEZ and LLAP

engines successfully. These 38 queries are randomly selected from the four broad classes of reporting, ad-

hoc, iterative OLAP, and data mining queries of 99 TPC-DS queries.

Total runtime comparison of 38 queries

0

500

1000

1500

2000

2500

3000

3500

Total runtime (s) (lower is better)

Hive/TEZ Hive/LLAP Uncached Hive/LLAP Cached

Worker nodes: 9 File format: ORC Configuration: Default HDP 3.0.1



As shown, LLAP delivers a dramatic performance gain. The aggregated runtime of 38 queries on the

Hive/TEZ and Hive/LLAP engines has a substantial 2x performance boost in the case of uncached

Hive/LLAP, and a 12x decrease in times for cached Hive/LLAP.

Note: Hive on LLAP uncached is defined as when the compute resource memory is empty, and data is pulled

from the Isilon system for the first time. Hive LLAP cached is defined as when the compute resource memory

is populated with the dataset during interactive querying.

Figure 5 shows Hive LLAP performance compared to Hive TEZ for the database stored on the Isilon OneFS

system. In this case, Hive on LLAP ran uncached (with an empty compute resource cache), and the average

speed increase is 2x, proving that Isilon OneFS supports Hive LLAP with a performance boost compared to

Hive on the TEZ engine.

Hive/LLAP uncached shows a 2x runtime performance boost compared to Hive/TEZ

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0

50

100

150

200

250

query

1.s

ql

query

3.s

ql

query

6.s

ql

query

8.s

ql

query

12.s

ql

query

15.s

ql

query

20.s

ql

query

21.s

ql

query

22.s

ql

query

26.s

ql

query

30.s

ql

query

32.s

ql

query

33.s

ql

query

34.s

ql

query

37.s

ql

query

39.s

ql

query

40.s

ql

query

41.s

ql

query

42.s

ql

query

43.s

ql

query

45.s

ql

query

48.s

ql

query

52.s

ql

query

53.s

ql

query

55.s

ql

query

56.s

ql

query

60.s

ql

query

66.s

ql

query

68.s

ql

query

71.s

ql

query

73.s

ql

query

79.s

ql

query

83.s

ql

query

84.s

ql

query

85.s

ql

query

86.s

ql

query

90.s

ql

query

91.s

ql

query

98.s

ql

Spped incre

ase (

TE

Z/L

LA

P)

(big

ger

is b

etter)

Runtim

e (

s)

(log.

scale

) (low

er

is b

etter) TPC-DS queries,

3TB, 9 nodes

Hive/TEZ (s) Hive/LLAP Uncached Speedup

2x average speed increase



Figure 6 displays the performance for Hive LLAP cached (the compute resource cached data set from the

previous executions). This shows a 13x performance boost compared to Hive on the TEZ engine. Hive LLAP

data is cached during interactive querying on the same dataset, and the performance drastically increases in

Hive LLAP due to the hot cache on compute nodes. This minimizes the data transfer between storage and

compute, and enables sub-second queries in Hive by keeping all data and servers running and in-memory all

the time, while retaining to ability for elastic scalability within the YARN cluster.

Hive/LLAP cached shows a 13x runtime performance boost compared to Hive/TEZ

LLAP, along with Apache Ranger, enables fine-grained security for the Hadoop ecosystem, including data

masking and filtering, by providing interfaces for external clients like Spark to read. LLAP is ideal for Hadoop

clusters with decoupled compute and a storage-like cloud because it caches data in memory and keeps it

compressed, overcoming long cloud-storage access times and stretching the amount of data that can fit the

compute cluster memory.

0

5

10

15

20

25

30

35

0

50

100

150

200

250query

1.s

ql

query

3.s

ql

query

6.s

ql

query

8.s

ql

query

12.s

ql

query

15.s

ql

query

20.s

ql

query

21.s

ql

query

22.s

ql

query

26.s

ql

query

30.s

ql

query

32.s

ql

query

33.s

ql

query

34.s

ql

query

37.s

ql

query

39.s

ql

query

40.s

ql

query

41.s

ql

query

42.s

ql

query

43.s

ql

query

45.s

ql

query

48.s

ql

query

52.s

ql

query

53.s

ql

query

55.s

ql

query

56.s

ql

query

60.s

ql

query

66.s

ql

query

68.s

ql

query

71.s

ql

query

73.s

ql

query

79.s

ql

query

83.s

ql

query

84.s

ql

query

85.s

ql

query

86.s

ql

query

90.s

ql

query

91.s

ql

query

98.s

ql

Speed incre

ase (

TE

Z/L

LA

P)

(big

ger

is b

etter)

Runtim

e (

s)

(log.

scale

) (low

er

is b

etter) TPC-DS queries,

3 TB, 9 nodes

Hive/TEZ (s) Hive/LLAP Cached SpeedUp

12.65x average speed increase

Conclusion


6 Conclusion The Hortonworks Hive LLAP has been designed to enable sub-second SQL analytics on Hadoop in

anticipation of vast increases in data volumes. The ability of OneFS to scale to multiple petabytes in a single

file system while delivering high performance I/O makes Isilon storage ideal for Hive, and offers optimal

workload portability if considering offloading workloads from a legacy EDW.

This paper shows that the Isilon F800 all-flash NAS storage solution meets Hive LLAP SQL compliance, and

performs very well under I/O load with a Hive Data warehouse for both small and large data sets. The

Hortonworks Hive Testbench results for all 99 TPC-DS queries are included in this paper and show excellent

results in terms of execution time for a 3 TB scale factor.

With Dell EMC Isilon storage, enterprise organizations and Hadoop/Hive administrators can effortlessly scale

from tens of terabytes to tens of petabytes within a single file system, single volume, and with a single point of

administration. Isilon storage delivers high performance and high throughput without adding management

complexity.

Configuration details


A Configuration details

A.1 Hadoop cluster configurations

Figure 7 shows the HDP 3.0.1 Hadoop cluster configuration and software version used in testing, and Figure

8 shows the Apache Ambari™ server version.

Hadoop cluster software versions

Apache Ambari server version



A.1.1 OneFS service configurations

Isilon OneFS is presented as a service to the HDP 3.0.1 Hadoop cluster. Figure 9 and Figure 10 show the

configuration set used in testing.

OneFS service advanced core-site.xml file configurations

OneFS service hdfs-site.xml and custom core-site.xml file configurations



A.1.2 YARN service configurations

Figure 11 and Figure 12 show the HDP 3.0.1 cluster YARN service configurations set used in testing.

Yarn service general configurations



Yarn configuration summary



A.1.3 Hive service configurations

Figure 13 through Figure 17 show the HDP 3.0.1 Hadoop cluster Hive service configuration set used in

testing.

TEZ general configurations



TEZ -site.xml configurations, all set to default

Hive service summary



Hive configuration setting



Hive configuration setting 2



A.1.4 YARN queue manager

Figure 18 shows the HDP 3.0.1 Hadoop cluster YARN queue manager configured used in testing.

LLAP queue setting

A.2 Isilon configuration

The following features were configured on the Isilon cluster. The Smart features shown are product

differentiators that significantly enhance data storage performance and resiliency.

• Enable SmartPools settings across all Isilon nodes.

• Enable SmartConnect to provide automatic client connection load-balancing and failover capabilities.

• Enable SmartCache for write performance and Streaming Access for Data Access Optimization.

• Use optimization for a streaming data-access pattern.

• Use 40 Gb/s external network ports for NFS connections and internal 40 Gb/s ports for the data

interconnect network.

• Increase network MTU to 9000 (jumbo frames) for both internal and external networks.



The storage pool is created on the SmartPools tab, allowing you to specify the Isilon nodes and protection

settings. Figure 19 shows the storage pool configured for the Hadoop and Isilon cluster.

Storage Pools setting

Figure 20 shows the internal and external network configuration set in the network configuration tab, where

you can specify MTU size, IP info, and DNS server information.

Isilon network settings



The SmartConnect information is configured within the pool properties. In the Hadoop Isilon cluster, the pool

name is pool0 and the SmartConnect information (with IP and DNS info blacked out) is shown in Figure 21

Isilon pool details

Note: Isilon storage provides two 40 GbE front-end and two 40GbE back-end ports with each node. The

Hadoop and Isilon cluster was only configured with one 40 GbE front-end port and one 40GbE back-end port

on each node during performance testing. With production deployments, use both front-end and back-end 40

GbE ports.



A.2.1 OneFS HDFS settings

Figure 22 shows the HDFS is configured to Isilon system zone, namely /ifs/hdfs-hdp301. The root hdfs path

was tested in this paper.

OneFS HDFS settings



A.2.2 OneFS TCP tuning

The default TCP stack of OneFS requires tuning for Hadoop and 40 GbE connectivity. The tuning needs to be

done within the CLI directly on Isilon. A tcptune.sh script is available at GitHub.

Run sh ./tcptune.sh Max to make the changes. An example script run is shown as follows:

Before changes:

isilon# sh ./tcptune.sh Max

Tuning TCP stack to Max

TCP sysctls before...

kern.ipc.maxsockbuf=2097152

net.inet.tcp.sendbuf_max=2097152

net.inet.tcp.recvbuf_max=2097152

net.inet.tcp.sendbuf_inc=8192

net.inet.tcp.recvbuf_inc=16384

net.inet.tcp.sendspace=131072

net.inet.tcp.recvspace=131072

efs.bam.coalescer.insert_hwm=209715200

efs.bam.coalescer.insert_lwm=178257920

After changes:

Apply tuning...

Value set successfully








TCP sysctls after...

kern.ipc.maxsockbuf=104857600

net.inet.tcp.sendbuf_max=52428800

net.inet.tcp.recvbuf_max=52428800

net.inet.tcp.sendbuf_inc=16384

net.inet.tcp.recvbuf_inc=32768

net.inet.tcp.sendspace=26214400

net.inet.tcp.recvspace=26214400

efs.bam.coalescer.insert_hwm=209715200

efs.bam.coalescer.insert_lwm=178257920

net.inet.tcp.mssdflt=8948

https://github.com/bonibruno/OneFS-HDFS-Tools/blob/master/tcptune.sh

Technical support and resources


B Technical support and resources

Dell.com/support is focused on meeting customer needs with proven services and support.

Storage technical documents and videos provide expertise that helps to ensure customer success on Dell

EMC storage platforms.

B.1 Related resources

See the following referenced or recommended resources:

• Hortonworks Hive LLAP

• Hortonworks Hive TPC-DS

• Hortonworks HDP 3.0.1

• Isilon technical Overview

• Hortonworks HDP 3.0.1 Install on Isilon OneFS8.1.2

http://www.dell.com/support

https://www.dell.com/storageresources

https://hortonworks.com/blog/announcing-apache-hive-2-1-25x-faster-queries-much/

https://github.com/hortonworks/hive-testbench

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/index.html

http://www.emc.com/collateral/hardware/white-papers/h10719-isilon-onefs-technical-overview-wp.pdf

https://support.emc.com/docu90563_Isilon-OneFS-8.1-2-with-Hadoop-and-Hortonworks-for-Kerberos-Installation-Guide.pdf