polybase in actionfiles.informatandm.com/uploads/2018/10/polybase_in... · 2018-10-10 · • sql...

#ITDEVCONNECTIONS | ITDEVCONNECTIONS.COM

Polybase In Action


Kevin FeaselEngineering Manager, Predictive Analytics

ChannelAdvisor


Who Am I? What Am I Doing Here?

Catallaxy Services

Curated SQL

We Speak Linux@feaselkl

https://www.catallaxyservices.com/

https://curatedsql.com/

https://wespeaklinux.com/

https://www.twitter.com/feaselkl


Polybase

Polybase is Microsoft's newest technology for connecting to remote servers.

It started by letting you connect to Hadoop and has expanded since then to include Azure Blob Storage. Polybase is also the best method to load data into Azure SQL Data Warehouse.


Polybase Targets

• SQL Server to Hadoop (Hortonworks or Cloudera, on-prem or IaaS)

• SQL Server to Azure Blob Storage• Azure Blob Storage to Azure SQL Data Warehouse

In all three cases, you can use the T-SQL you know rather than a similar SQL-like language (e.g., HiveQL, SparkSQL, etc.) or some completely different language.


Polybase Targets – SQL Server 2019

• SQL Server to SQL Server• SQL Server to Oracle• SQL Server to MongoDB• SQL Server to Teradata• SQL Server to ODBC (e.g., Spark)


Massive Parallel Processing

Polybase extends the idea of Massively Parallel Processing (MPP) to SQL Server. SQL Server is a classic "scale-up" technology: if you want more power, add more RAM/CPUs/resources to the single server.

Hadoop is a great example of an MPP system: if you want more power, add more servers; the system will coordinate processing.


Why MPP?

It is cheaper to scale out than scale up: 10 systems with 256 GB of RAM and 8 cores is a lot cheaper than a system with 2.5 TB of RAM and 80 cores.At the limit, you eventually run out of room to scale up, but scale out is much more practical: you can scale out to 2 petabytes of RAM but good luck finding a single server that supports this amount!There is additional complexity involved, but MPP systems let you move beyond the power of a single server.


Polybase As MPP

MPP requires a head node and 1 or more compute nodes. Polybase lets you use SQL Servers as the head and compute nodes. Scale-out servers must be on an Active Directory domain. The head node must be Enterprise Edition, though the compute nodes can be Standard Edition.Polybase lets SQL Server compute nodes talk directly to Hadoop data nodes, perform aggregations, and then return results to the head node. This removes the classic SQL Server single point of contention.


Timeline

• Introduced in SQL Server Parallel Data Warehouse (PDW) edition, back in 2010

• Expanded in SQL Server Analytics Platform System (APS) in 2012.

• Released to the "general public" in SQL Server 2016, with most support being in Enterprise Edition.

• Extended support for additional technologies (like Oracle, MongoDB, etc.) will be available in SQL Server 2019.


Motivation

Today's talk will focus on using Polybase to integrate SQL Server 2016/2017 with Hadoop and Azure Blob Storage.

We will use a couple smaller data sources to give you an idea of how Polybase works. Despite the size of the demos, Polybase works best with a significant number of compute nodes and Hadoop works best with a significant number of data nodes.


Installation Pre-Requisites

1. SQL Server 2016 or later, Enterprise or Developer Edition2. Java Runtime Environment 7 Update 51 or later (get the

latest version of 8 or 9; using JRE 9 requires SQL Server 2017 CU4)

3. Machines must be on a domain if you want to use scale-out

4. Polybase may only be installed once per server. If you have multiple instances, choose one. You can enable on multiple VMs, however.


Installation

Select the “New SQL Server stand-alone installation” link in the SQL Server installer:


Installation

When you get to feature selection, check the “PolyBase Query Service for External Data” box:


Installation

If you get the following error, you didn’t install the Java Runtime Environment.

If you have JRE 9, you need SQL Server 2017 CU4 or later for SQL Server to recognize this.


Installation

For standalone installation, select the first radio button. This selection does not require your machine be connected to a domain.


Installation

The Polybase engine and data movement service accounts are NETWORK SERVICE accounts by default. There are no virtual accounts for Polybase.


Installation

After installation is complete, run the following against the SQL Server instance:

sp_configure @configname = 'hadoop connectivity', @configvalue = 7;GORECONFIGUREGO

Set the value to 6 for Cloudera’s Hadoop distribution, or 7 for Hortonworks or Azure Blob Storage.


Hadoop Configuration

First, we need to make sure our Hadoop and SQL Server configuration settings are in sync.

We need to modify the yarn-site.xml and mapred-site.xml configuration files.

If you do not do this correctly, then MapReduce jobs will fail!



You will need to find your Hadoop configuration folder that came as part of the Polybase installation. By default, that is at:

C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Binn\Polybase\Hadoop\conf

Inside this folder, there are two files we care about.



Next, go looking for your Hadoop installation directory. On HDP, you'll find it at:

/usr/hdp/[version]/hadoop/conf/

Note that the Polybase docs use /usr/hdp/current, but this is a bunch of symlinks with the wrong directory structure.



Modify yarn-site.xml and change the yarn.application.classpath property. For the Hortonworks distribution of Hadoop (HDP), you’ll see a series of values like:

<value>/usr/hdp/2.4.3.0-227/hadoop/*,/usr/hdp/2.4.3.0-227/hadoop/lib/*, …</value>

Replace 2.4.3.0-227 with your HDP version.



Include the following snippet in your mapred-site.xml file:<property><name>yarn.app.mapreduce.am.staging-dir</name><value>/user</value></property><property><name>mapreduce.jobhistory.done-dir</name><value>/mr-history/done</value></property><property><name>mapreduce.jobhistory.intermediate-done-dir</name><value>/mr-history/tmp</value></property>

Without this configured, you will be unable to perform MapReduce operations on Hadoop.


Polybase Basics

In this section, we will look at three new constructs that Polybase introduces: external data sources, external file formats, and external tables.


External Data Source

External data sources allow you to point to another system. There are several external data sources, and we will look at two today.


External Data Source

CREATE EXTERNAL DATA SOURCE [HDP] WITH(

TYPE = HADOOP,LOCATION = N'hdfs://sandbox.hortonworks.com:8020',RESOURCE_MANAGER_LOCATION =

N'sandbox.hortonworks.com:8050')

The LOCATION is the NameNode port and is needed for Hadoop filesystem operations. RESOURCE_MANAGER_LOCATION is the YARN port and is needed for predicate pushdown.


External File Format

External file formats explain the structure of a data set. There are several file formats available to us.


External File Format: Delimited File

Delimited files are the simplest to understand but tend to be the least efficient.CREATE EXTERNAL FILE FORMAT file_format_nameWITH (

FORMAT_TYPE = DELIMITEDTEXT [ , FORMAT_OPTIONS ( <format_options> [ ,...n ] ) ] [ , DATA_COMPRESSION = {

'org.apache.hadoop.io.compress.GzipCodec' | 'org.apache.hadoop.io.compress.DefaultCodec'

} ]);


External File Format: Delimited File

<format_options> ::= {

FIELD_TERMINATOR = field_terminator| STRING_DELIMITER = string_delimiter| DATE_FORMAT = datetime_format| USE_TYPE_DEFAULT = { TRUE | FALSE }

} </format_options>


External File Format: RCFile

Record Columnar files are an early form of columnar storage.

CREATE EXTERNAL FILE FORMAT file_format_nameWITH (

FORMAT_TYPE = RCFILE, SERDE_METHOD = {

'org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe' | 'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'

} [ , DATA_COMPRESSION =

'org.apache.hadoop.io.compress.DefaultCodec' ]);


External File Format: ORC

Optimized Row Columnar files are strictly superior to RCFile.


FORMAT_TYPE = ORC [ , DATA_COMPRESSION = {

'org.apache.hadoop.io.compress.SnappyCodec' | 'org.apache.hadoop.io.compress.DefaultCodec' }

]);


External File Format: Parquet

Parquet files are also columnar. Cloudera prefers Parquet, whereas Hortonworks prefers ORC.


FORMAT_TYPE = PARQUET [ , DATA_COMPRESSION = {

'org.apache.hadoop.io.compress.SnappyCodec' | 'org.apache.hadoop.io.compress.GzipCodec' }

]);


External File Format: ComparisonMethod Good Bad Best Uses

Delimited Easy to use Less efficient, slower performance

Easy Mode

RC File Columnar Strictly superior options Don’t use this

ORC Great agg perf Columnar not always a good fit; slower to write

Non-nested files with aggregations of subsets of columns

Parquet Great agg perf Columnar not always a good fit; often larger than ORC

Nested data


External Tables

External tables use external data sources and external file formats to point to some external resource and visualize it as a table.


External Tables

CREATE EXTERNAL TABLE [dbo].[SecondBasemen](

[FirstName] [VARCHAR](50) NULL,[LastName] [VARCHAR](50) NULL,[Age] [INT] NULL,[Throws] [VARCHAR](5) NULL,[Bats] [VARCHAR](5) NULL

)WITH(

DATA_SOURCE = [HDP],LOCATION = N'/tmp/ootp/secondbasemen.csv',FILE_FORMAT = [TextFileFormat],REJECT_TYPE = VALUE,REJECT_VALUE = 5

);


External Tables

External tables appear to end users just like normal tables: they have a two-part schema and even show up in Management Studio, though in an External Tables folder.


Demo Time


Querying Hadoop

Once we have created an external table, we can write queries against it just like any other table.


Demo Time


Querying Hadoop – MapReduce

In order for us to be able to perform a MapReduce operation, we need the external data source to be set up with a resource manager. We also need one of the two:

1. The internal cost must be high enough (based on external table statistics) to run a MapReduce job.

2. We force a MapReduce job by using the OPTION(FORCE EXTERNALPUSHDOWN) query hint.

Note that there is no "cost threshold for MapReduce," so the non-forced decision is entirely under the Polybase engine's control.


Querying Hadoop – MapReduce

Functionally, MapReduce queries operate the same as basic queries. Aside from the query hint, there is no special syntax for MapReduce operations and end users don't need to think about it.

WARNING: if you are playing along at home, your Hadoop sandbox should have at least 12 GB of RAM allocated to it. This is because Polybase creates several 1.5 GB containers on top of memory requirements for other Hadoop services.


Demo Time


Querying Hadoop – Statistics

Although external tables have none of their data stored on SQL Server, the database optimizer can still make smart decisions by using statistics.



Important notes regarding statistics:

1. Stats are not auto-created.2. Stats are not auto-updated.3. The only way to update stats is to drop and re-create the

stats.4. SQL Server generates stats by bringing the data over, so

you must have enough disk space! If you sample, you only need to bring that percent of rows down.



Statistics are stored in the same location as any other table's statistics, and the optimizer uses them the same way.


Demo Time


Querying Hadoop – Data Insertion

Not only can we select data from Hadoop, we can also write data to Hadoop.

We are limited to INSERT operations. We cannot update or delete data using Polybase.


Demo Time


Azure Blob Storage

Hadoop is not the only data source we can integrate with using Polybase. We can also insert and read data in Azure Blob Storage.

The basic constructs of external data source, external file format, and external table are the same, though some of the options are different.


Azure Blob Storage

Create an external data source along with a scoped database credential (for secure access to this blob):CREATE MASTER KEY ENCRYPTION BY PASSWORD = '{password}';GOCREATE DATABASE SCOPED CREDENTIAL AzureStorageCredentialWITH IDENTITY = 'cspolybase',SECRET = '{access key}';GOCREATE EXTERNAL DATA SOURCE WASBFlightsWITH (

TYPE = HADOOP, LOCATION = 'wasbs://[email protected]',CREDENTIAL = AzureStorageCredential

);


Azure Blob Storage

External file formats are the same between Hadoop and Azure Blob Storage.

CREATE EXTERNAL FILE FORMAT [CsvFileFormat] WITH(

FORMAT_TYPE = DELIMITEDTEXT,FORMAT_OPTIONS(

FIELD_TERMINATOR = N',',USE_TYPE_DEFAULT = True

));


Azure Blob Storage

External tables are similar to Hadoop as well.CREATE EXTERNAL TABLE [dbo].[Flights2008](...)WITH(

LOCATION = N'historical/2008.csv.bz2',DATA_SOURCE = WASBFlights,FILE_FORMAT = CsvFileFormat,-- Up to 5000 rows can have bad values before Polybase

returns an error.REJECT_TYPE = Value,REJECT_VALUE = 5000

);


Azure Blob Storage

Start with a set of files:


Azure Blob Storage

After creating the table, select top 10:


Azure Blob Storage

Running an expensive aggregation query:

SELECTfa.[year],COUNT(1) AS NumberOfRecords

FROM dbo.FlightsAll faGROUP BY

fa.[year]ORDER BY

fa.[year];


Azure Blob Storage

While we're running the expensive aggregation query, we can see that the mpdwsvc app chews up CPU and memory:


Azure Blob Storage

Create a table for writing to blob storage:

CREATE EXTERNAL TABLE [dbo].[SecondBasemenWASB](...)WITH(

DATA_SOURCE = [WASBFlights],LOCATION = N'ootp/',FILE_FORMAT = [CsvFileFormat],REJECT_TYPE = VALUE,REJECT_VALUE = 5

)


Azure Blob Storage

Insert into the table:

INSERT INTO dbo.SecondBasemenWASBSELECT

sb.FirstName,sb.LastName,sb.Age,sb.Bats,sb.Throws

FROM Player.SecondBasemen sb;


Azure Blob Storage

Eight files are created:


Azure Blob Storage

Multiple uploadscreate separatefile sets:


Other Azure Offerings – Azure SQL DW

Polybase features prominently in Azure SQL Data Warehouse, as Polybase is the best method for getting data into an Azure SQL DW cluster.



Access via SQL Server Data Tools,not SSMS:



Once connected, we cansee the database.



Create an external data source to Blob Storage:CREATE MASTER KEY ENCRYPTION BY PASSWORD = '{password}';GOCREATE DATABASE SCOPED CREDENTIAL AzureStorageCredentialWITH IDENTITY = 'cspolybase',SECRET = '{access key}';GOCREATE EXTERNAL DATA SOURCE WASBFlightsWITH (

TYPE = HADOOP,LOCATION =

'wasbs://[email protected]',CREDENTIAL = AzureStorageCredential

);



Create an external file format:

CREATE EXTERNAL FILE FORMAT [CsvFileFormat] WITH(

FORMAT_TYPE = DELIMITEDTEXT,FORMAT_OPTIONS(

FIELD_TERMINATOR = N',',USE_TYPE_DEFAULT = True

));GO



Create an external table:CREATE EXTERNAL TABLE [dbo].[Flights2008] (...)WITH(

LOCATION = N'historical/2008.csv.bz2',DATA_SOURCE = WASBFlights,FILE_FORMAT = CsvFileFormat,-- Up to 5000 rows can have bad values before

Polybase returns an error.REJECT_TYPE = Value,REJECT_VALUE = 5000

);



Blob Storage data retrieval isn’t snappy:



Use CTAS syntax to create an Azure SQL DW table:

CREATE TABLE[dbo].[Flights2008DW]WITH(

CLUSTERED COLUMNSTORE INDEX,DISTRIBUTION = HASH(tailnum)

)AS SELECT * FROM dbo.Flights2008;GO



Azure SQL Data Warehouse data retrieval is snappy:



We can export data to Azure Blob Storage:CREATE EXTERNAL TABLE [dbo].[CMHFlights]WITH(

LOCATION = N'columbus/',DATA_SOURCE = WASBFlights,FILE_FORMAT = CsvFileFormat,-- Up to 5000 rows can have bad values before Polybase

returns an error.REJECT_TYPE = Value,REJECT_VALUE = 5000

)AS SELECT * FROM dbo.FlightsAllDW WHERE dest = 'CMH';GO



This CETAS syntax lets us write out the result set:



CETAS created 60 files,1 for each Azure DWcompute node:


Other Azure Offerings – Data Lake

Polybase can only read from Azure Data Lake Storage if you are pulling data into Azure SQL Data Warehouse.

The general recommendation for SQL Server is to use U-SQL and Azure Data Lake Services to pull data someplace where SQL Server can read the data.


Other Azure Offerings – HDInsight

Polybase is not supported in Azure HDInsight. Polybase requires access to ports that are not available in an HDInsight cluster.

The general recommendation is to use Azure Blob Storage as an intermediary between SQL Server and HDInsight.


Other Azure Offerings – SQL DB

Polybase concepts like external tables drive Azure SQL Database's cross-database support.

Despite this, we can not use Polybase to connect to Hadoop or Azure Blob Storage via Azure SQL Database.


Issues -- Docker

Polybase has significant issues connecting to DockerizedHadoop nodes. For this reason, I do not recommend using the HDP 2.5 or 2.6 sandboxes, either in the Azure marketplace or on-prem.

Instead, I recommend building your own Hadoop VM or machine using Ambari.


Issues -- MapReduce

Current-gen Polybase supports direct file access and MapReduce jobs in Hadoop. It does not support connecting to Hive warehouses, using Tez, or using Spark.

Because Polybase's Hadoop connector does not support these, it must fall back on a relatively slow method for data access.


Issues– File Formats

Polybase only supports files without in-text newlines. This makes it impractical for parsing long text columns which may include newlines.

Polybase is limited in its file format support and does not support the Avro file format, which is a superior rowstore data format.


Big Data Clusters – Next-Gen Polybase

The next generation of Polybase involves being able to connect to Oracle, Elasticsearch, MongoDB, Teradata, and anything with an ODBC interface (like Spark).

This is now available in the SQL Server 2019 public preview. The goal is to make SQL Server a “data hub” for interaction with various technologies and systems, with SQL Server as a virtualization layer.


For Further Thought…

Some interesting uses of Polybase:

• Hot/Cold partitioned views• Hadoop-based data lake enriched by SQL data• "Glacial" data in Azure Blob Storage• Replacement for linked servers (with Polybase vNext)


Wrapping Up

Polybase was one of the key SQL Server 2016 features. There is still room for growth (and a team hard at work), but it is a great integration point between SQL Server and Hadoop / Azure Blob Storage.

To learn more, go here: https://CSmore.info/on/polybase

And for help, contact me: [email protected] | @feaselkl

https://csmore.info/on/polybase

mailto:[email protected]

https://www.twitter.com/feaselkl

polybase in actionfiles.informatandm.com/uploads/2018/10/polybase_in... · 2018-10-10 · • sql...

Documents